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ABSTRACT 

This handbook has been written to enable schools and 
community agency staff to carry out required evaluations under the 
Drug-Free Schools and Communities Act. However, its applicability is 
not restricted to programs supported through that Act. The handbook 
describes the why and how of program evaluation and outlines the 
steps in conducting evaluations. A premise guiding this handbook is 
that many evaluations that use simple designs can be conducted 
without formal training in program evaluation. This handbook has 
three chapters. Chapter 1 is an overview of evaluation planning. 
Chapter 2 provides more detail on the steps in designing an 
evaluation, and chapter 3 tells the story of an evaluation conducted 
by a fictitious school district. The handbook presents the basic ' 
concepts that guide program evaluation. Where greater help may be 
needed, the discussion refers the reader to the appendix and to more 
detailed information from other sources. The guide also indicates 
points in the course of designing and carrying out an evaluation 
where program officials may wish to consult with evaluation 
specialists inside or outside their districts or organizations. An 
evaluation can be an important tool in improving the quality of a 
prevention program if it is integrated into the fabric of a program 
rather than added on after the fact. Program personnel are more 
likely to use the results of an evaluation when they play a role in 
deciding what to examine, conducting the evaluation, and interpeting 
the results. Many of the evaluation steps outlined in this handbook 
can be carried out by program staff in schools and community 
agencies . (Author) 
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introduction to Evaluation 



This handbook has been written to enable school and community agency staff to 
cany out required evaluations under the Drug-Free Schools and Communities Act (DFSCA). 
However, its applicability is not restricted to programs supported through that Act. The 
handbook describes the why and how of program evaluation and outlines the steps in 
conducting evaluations. A premise guiding this handbook is that many evaluations that use 
simple designs can be conducted without formal training in program evaluation. 

The hanubook has three chapters. Chapter 1 is an overview of evaluation planning. 
Chapter 2 provides more detail on the steps in designing an evaluation, and Chapter 3 tells 
the story of an evaluation conducted by a fictitious school district. The handbook presents 
the basic concepts that guide program evaluation. Where greater help may be needed, the 
discussion refers the reader to the appendix and to more detailed information from other 
sources. The guide also indicates points in the course of designing and carrying out an 
evaluation where program officials may Vvish to consult with evaluation specialists inside or 
outside their districts or organizations. 

An evaluation can be an important tool in improving the quality of a prevention 
program if it is integrated into the fabric of a program rather than added on after the fact. 
Program personnel are more likely to use the results of an evaluation when they play a role 
in deciding what to examine, conducting the evaluation, and interpreting the results. Many 
of the evaluation steps outlined in this handbook can be carried out by program staff in 
schools and community' agencies. 
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Why EvduQte Drug and Alcohol Prevention Projects? 



Prevention programs that address dmg and alcohol »5se are operating in a relatively 
new field. There are few interventions of proven effectiveness and the knowledge base is 
still growing. Thus, there are many reasons to conduct evaluations, including the following: 

♦ To determine the effectiveness of programs fo: participants; 

♦ To document that program objectives have been met; 

♦ To provide information about service delivery that will be useful 
to program staff and other audiences: and 

♦ To enable program staff to make changes that improve 
program effectiveness. 

In other words, evaluations help to foster accountability, determine whether programs "make 
a difference, ' and give staff the information they need to improve ser/ice delivery. 

In addition, the prevention programs supported through the DFSCA are required to 
assess their activities and services. All grant programs funded under DFSCA must conduct 
evaluations, including programs funded through: 

- State and local formula grants: 

- Federal activities grants: 

- School personnel training grants: 

- Counselor training grants: 

- iModel demonstration grants: and 

- Emergency grants. 

The legal requirement reflects the need for Federal accountability in administering the 
DFSCA. The U.S. Department of Education must report to Congress on the effectiveness of 
the DFSCA in establishing prevention programs for grades K-12 and in reducing dnig and 
alcohol use. The evaluations conducted by grantees will assist in that process. Evaluation 
can help expand practitioners* and policymakers' understanding of the effectiveness of 
DFSCA-supported programs. 
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This handbook will describe a \'ariety of e\'aluation activities so that school districts 
and communit\' agencies can tailor evaluations to their local program objectives and needs. 
For example, districts or agencies with limited e\'aluation resources may want to concentrate 
on finding out how effectively they are delivering ^V.e services that they set out to offer. An 
agency with restrictions on the services it can provide may want to know how those 
restrictions affect program deliver^^ 

The staff of districts or agencies with greater resources and evaluation capability can 
expand their evaluations to learn how successfully they are affecting student behavior and 
then build on their projects' most successful components. Districts may also have obligations 
10 report to local authorities or other constituencies on their programs' impact on student 
alcohol or other drug use. The most compelling argument for continuing a program is that it 
made a positive difference for participants and for a community. 




Whot is Evoluation? 



Evaluation is the systematic collection and analysis of data needed to make 
decisions, a process in which most well-run programs engage from the outset. Here are just 
some of the evaluation activities that are already likely to be incorporated into many 
programs or that can be added easily: 

♦ Pinpointing the services needed— for example, finding out what knowledge, 
skills, attitudes, or problem behaviors a drug or alcohol prevention program 
should address; 

♦ Establishing program objectives and deciding the particular evidence (such 
as the specific knowledge, attitudes, or behavior) that will demonstrate that 
the objectives have been met. A key to successful evaluation is a set of 
clear, measurable, and realistic program objectives. If objectives are 
unrealistically optimistic or are not measurable, the program may not be able 
to demonstrate that it has been successful even if it has done a good job; 

♦ Developing or seleaing from among alternative program approaches— for 
example, trying different curricula or policies and determining which ones 
best achieve the goals; 

♦ Tracking program objectives— for example, setting up a system that shows 
who gets services, how much service is delivered, how participants rate the 
services they receive, and which approaches are most readily adopted by 
staff; or 

♦ Trying out and assessing new program designs— determining the extent to 
which a particular approach is being implemented faithfully by school or 
agency personnel or the extent to which it attracts or retains participants. 

Through these types of activities, those who provide or administer services determine 
what to offer and boiv well they are offering those services. In addition, evaluation 
in drug education can identify program effects, helping staff and others to finJ out 
whether their programs have an impact on participants' knowledge or attitudes about 
drugs and alcohol, forestall participants' use of drugs, or reduce drug use. 

The different dimension:? of evaluation have formal names: process, outcome, 
and impact evaluation These three dimensions can also be thought of as a set of 
assessment options that build upon one another, allowing program staff to increase their 
knowledge abou( the activities they undertake as they incorporate more options or 
dimensions mto their evaluation. 
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Process evaluation describes and assesses program materials and 
activities. Examination of materials is likely to occur while programs are being developed, 
as a check on the appropriateness of the approach and procedures that will be used in the 
program. For example, program staff might systematically review the units in a curriculum to 
determine whether they adequately address all of the behaviors the program seeks to 
influence. A program administrator might observe teachers using the program and write a 
descriptive account of how students respond, then provide feedback to instructors. 

Examining the implementation of program activities is an important form of process 
evaluation, hnplementation analysis documents what actually transpires in a program and 
how closely it resembles the program s goals. For example, after a new drug-free school 
policy has been adopted, how is it enforced? If the policy mandates parent conferences for 
all first infractions and suspensions for subsequent infractions, is the policy heeded? If not. 
why? What could be done to achieve better enforcement? Establishing the extent and nature 
of program implementation is also an important first step in studying program outcomes; 
that is. it describes the inter\'entions to 
which any findings about outcomes 
may be attributed. 

Outcome evaluation 
assesses program achievements and 
ejects. Outcome evaluations suidy 
the immediate or direct effects of the 
program on participants. For example, 
when a lO-session program aimed at 
teaching refusal skills is completed, can 
the participants demonstrate the skills 
successfully? This type of e\'aluation 
is not unlike what happens when a 
teacher administers a lest before and 
after a unit to make sure the students 
have learned the material. The scope 
of an outcome evaluation can extend 
beyond knowledge or altitudes, 
however, to examine the immediate 
behavioral effects of programs. 




Impact evaluation looks bey ond the immediate resuUs ofpoUcies, 
instruction, or services to identify longer-term as weU as unintended program 
effects. It may also examine what happens when several programs operate in unison. For 
example, an impact evaluation might examine whether a program's immediate positive 
effects on behavior were sustained over time. It might also look at whether the introduction 
of a community-wide prevention program with components administered by schools, 
agencies, and churches resulted in fewer teenage dnig-relaied arrests or deaths. 

Some school districts and com.munity agencies may limit their inquiry to process 
evaluation. Others may have the interest and the resources to pursue an examination of 
whether their activities are affecting panicipants and others in a positive manner (outcome or 
impact evaluation). The choices should be made based upon local needs, resources, 
and requirements. 

Regardless of the kind of evaluation, all evaluations use data collected in a 
systematic manner. These data may be quantitative— such as counts of program participants, 
amounts of counseling or other services received, or extent of dmg use. They also may be 
qualitative— such as descriptions of what iranspired at a series of counseling sessions or an 
expert's best judgment of the age-appropriateness of a skills training curriculum. Successflil 
evaluations often blend quantitative and qualitative data collection. The choice of which to 
use should be made with an understanding that there is usually more than one way to 
answer any given question. 



Before assessing a 
program, it is critical to 
consider who is most 
iikeiy to need and use 
the information that 
will be obtained end 
for what purposes. 



Why Conduct Program Evaluations? 

Evaluations serve many purposes. Before assessing 
a program, it is critical to consider who is most likely to need 
and use the information that will be obtained and for what 
purposes. Listed below are some of the most common 
reasons to conduct evaluiations. These reasons cut ;':ross the 
three types of evaluation just mentioned. The degree to 
which the perspectives of the most imponant potential users 
are incorporated into an evaluation design will determine the 
usefulness of the effort. 
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Evaluation for Project Management 

Administrators are often most interested in keeping track of program activities and 
documenting the nature and e>aent of service delivery. The type of information they seek to 
collect might be called a 'management information system" (MIS). An evaluation for project 
management monitors the routines of program operations. It can provide program staff or 
administrators with information on such items as participant characteristics, program 
activities, allocation of staff resources, or program costs. Analyzing information of this type (a 
kind of process evaluation) can help program staff to make short-term corrections-ensuring, 
for example, that planned program activities are conducted in a timely manner. This analysis 
can also help staff to plan future program direction-such as determining resource needs for 

the coming school year. 

Operations data are important for responding to information requests from 
constituents, such as funding agencies, school boards, boards of directors, or community 
leaders. Also, descriptive program data are one of the bases upon which assessments of 
program outcome are built-it does not make sense to conduct an outcome study if results 
can not be connected to specific program activities. An MIS also can keep track of students 
when the program ends to make future follow-up possible. 

Evaluation for Staying On Imk 

Evaluation can help to ensure that project activities continue to reflect project plans 
and goals Data collection for project management may be similar to data collection for 
staving on track, but more information might also be needed. An MIS could indicate how 
manv students participated in a prevention club meeting, but additional information would 
be needed to reveal why participants attended, what occurred at the meeting, how useful 
participants found the session, or what changes the club leader would recommend. This 
type of evaluation can help to strengthen service deliver)' and to maintain the connection 
lietween program goals, objectives, and services. 
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Evaluotion for Projed Efficiency 

Evaluation can help to streamline ser\-ice deliver\- or to enhance coordination among 
various program components, lowering the cost of ser\-ice. Increased efficiency can enable a 
program to ser\'e more people, offer more services, or target ser\-ices to those whose needs 
are greatest. Evaluation for program efficiency might focus on identifying the areas in which 
a program is most successful in order to capitalize upon them. It might also identif\- 
weaknesses or duplication in order to make improvements, eliminate some services, or refer 
participants to ser\-ices elsewhere. Evaluations of both program process and program 
outcomes are u.sed to determine efficiency. 

Evaluation for Project Accountobility 

When it comes to evaluation for accountability, the users of the e\-aluation results 
likely will come from outside of program operatii)ns: parent groups, funding agencies, 
elected officials, or other policymakers. Be it a process or an outcome e\-aluation. the 
methods used in accountability evaluation mu.st lie scientifically defensible, and able to 
.stand up to greater scmtiny than methods used in evaluations that are intended primarily for 
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"in-house" use. Vet even sophisticated evaluations must present results in ways that 
are understandable to lay audiences, because outside officials are not likely to be 
evaluation specialists. 



Evaluation for New Program Development and Dissemination 

Evaluating new approaches is verv' important to program development in any field. 
Developers of new programs designed to prevent drug and alcohol abuse need to conduct 
methodical evaluations of their efforts before making claims to potential users. Rigorous 
evaluation of longer-term program outcomes is a prerequisite to asserting that a new model 
is effective. School districts or community agencies that seek to disseminate their approaches 
to other potential users may wish to consult an evaluation specialist, perhaps a professor 
from a local university', in conducting this kind of evaluation. 



Risks of Evaluation 



Despite their value, evaluations are not always welcomed. Because they carry risks 
and use scarce resources, and because staff may be unsure how to conduct them, 
evaluations are often a low priority for programs. Evaluations are sometimes postponed until 
the last possible minute or avoided altogether. By understanding the potential difficulties 
before designing an evaluation, however, it is possible to avoid some of those risks or to 
minimize their effects. 

Evaluations can create anxiety among program 
staff. Staff members may feel threatened by an evaluation 
because they believe that their individual performance is 
being scrutinized or that the program's fate hangs in the 
balance. They may believe that the tools of evaluation are 
ill-suited to measure the positive changes they see 
occurring. The best method to o\*ercome staff members' 
fears and resistance is to involve them in designing the 
evaluation and in interpreting its findings. 



Staff will be less likely 
to see on evaluation 
as interference if they 
are consulted in its 
design and execution. 
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Evaluations can interfere with progmm activities. Sometimes there are trade-offs 
between program evaluation and service deliver\\ For example. obser\'ation of a counseling 
session by an outsider may interfere with the group process. Administration of question- 
naires may take time away from instruction or other activities. There are no simple solutions 
to these problems, but careful evaluation planning, limits on an evaluation s scope, and 
continued attention to its time and resource burden can minimize disruption. Again, if 
staff are consulted in its design and execution, they will be less likely to see an evaluation 
as interference. 

Evaluations compete with services for scarce resources. For example, a program 
may have to balance the cost of an evaluation specialist or computer time to process data 
against the cor^t of an additional counselor. Careful planning can reduce evaluation costs, 
however, and a solid evaluation may help to reduce program costs later by highlighting 
opportunities for program efficiency. 

Evaluation results may be misused. Care must be exercised in the interpretation of 
data in order to avoid exaggerated or unwarranted claims of program effectiveness. The 
inevitable loss of credibility' from such practices far outweighs any short-term gains. To 
forestall problems, it is important to make sure that results are stated clearly and 
unambiguously. Vaguely worded reports are more likely to be misinterpreted or misrepre- 
sented by others. A later section of the handbook discusses ways to present e\-aluation data 
clearly and fairly. 

Steps in Plonning Evaluotions 

Assuming that the benefits, risks, and costs ha\-e been considered and that the 
decision to proceed has been reached, there are practical steps in designing e\-aluations. 
This section outlines some of the decisions that school and community* prevention program 
staff must make in planning program evaluations. Chapters 2 and 3 of the handbook will 
discuss these steps in greater depth, using a fictitious school district evaluation to highlight 
major activities. 
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Identifying the Evduotlon's Consumers 

This point was made earlier, but it is imponant to reiterate that identifying the 
potential users will help to determine what questions are most imponant. v^'hat data will be 
viewed as credible, what analyses should be conduaed. and how results should be 
transmitted and displayed. It is a good idea to solicit the views of other consumers, along 
with program staff, in drawing up the evaluation questions. Depending on the programs and 
setting, consumers may range from parents and the school board to other institutions, 
funding agencies, or broadcast media. 

Choosing the important Evaluation Questions 

There is rarely enougli time or resources to answer all of the questions about 
program practice and effects that consumers pose. A way must be found to establish 
priorities and to limit the numl^er of questions. The most desirable method is to agree on a 
limited number of evaluation questions when the program goals and objectives are first 
established, but often the evaluation questions are drawn up after the fact or the program 
has multiple goals. Under these circum,stances. the numlx^r of possible questions may be 

'17 



One device to limit the 
inquiry is to osk eoch critical 
evaluotion user to complete 
the stotement, 1 need to 

know because I 

need to decide 



very* large. One device to limit the inquiry- is to ask 
each critical evaluation user to complete the statement, 
need to know because I need to decide 



As noted earlier, some programs may choose only 
questions that describe the intervention they carried 
out. while others may go further and examine the 
impact of the intervention. 



When the range of possible questions is 
agreed upon, estimates can be made about the benefits 
and costs of answering or not answering each of them. 
Some questions may be too costly to answer. Others 



may require a level of expertise in evaluation that is not available to the school district or 
agency. There may be simple measures that can achieve adequate results for some questions, 
making them candidates for inclusion, Fcpr example, a postcard follow-up surv'ey of teachers 
who attended an in-ser\'ice prograin might be sufficient to ask a few key questions about use 
of program materials. On the other hand, extensive questionnaires or classroom observation 
might be necessary to determine the extent to which, or the manner in which, teachers used 
particular instmctional techniques. 

Mopping Out an Evaluation Work Plan 

It is critical to create a step-by-step work plan for conducting the evaluation. The 
first step will be to review the questions and group them in some logical manner — by subject 
area, by the data needed to address them, by proces.s, outcome/ impact, or in some other 
manner. The plan should then outline the data that will be collected and how the 
information gathered will relate to each evaluation question. Suppose the questions 
are: how manv hours of instmction and practice in daig refusal skills did the typical .student 
receive during a semester, and did the number of hours var\^ substantially by classroom or 
school? What method(s) will be u.sed to document the extent and nature of the services 
provided? Will e\'aluators review .student records. revie^\ teacher logs or lesson plans. 
inter\-iew all the teachers or a .sample of teachers, or admini.ster student cjuestionnaircs? If 
teachers' les.son plans are used to document the amount of iastaiction, what data will be 
collected from those records and what evidence will indicate that refusal skill in.staiction 
look place on a particular day? What if le.s.son plans and teacher questionnaires disagree? 
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Once the data are in hand, how will the determination be made that substantial \'ariation in 
implementation has or has not occurred across schools? More detail on planning for data 
collection and analysis is presented in Chapter 2, 

^ Moking Sure Adequote Resources ore ot Hond to Corry Out All Functions 

Evaluation is labor intensive. At a minimum, an evaluation of a school or 
community prevention program will recjuire that the following resource-intensive functions 
be performed: 

♦ Evaluation Planning— formulating the o\'erall evaluation strategy and 
identif\'ing or developing the necessary- evaluation instruments. A study of 
outcomes' also may necessitate establishing participant and comparison 
groups and gaining parental consent for student questionnaires; 

♦ Data Collection— administering questionnaires, conducting interviews, 
obser\''ing program operations, or reviewing or entering data from existing 
data sources: 

♦ Data Coding— collating the information gained through data collection, 
ensuring that it is accurate, and translating colleaed data into usable formats 
for analysis: and 

♦ Data Analysis-Conducting any statistical analyses related to evaluation 
hypotheses, preparing summary' statistics, charts, tables, and graphs. 

i:>ata collection and coding may be performed by program staff or sometimes students (with 
proper training and supen-ision). but data analysis may require specialized skills and training. 
This is another activity for which program staff might wish to consult an outside expert. 

Even simple e\ aluation designs probably will require the use of computers to 
maintain an MIS or to aggregate data on participant and comparison groups, sen'ices. and 
the like. Computers probably will be used for data analysis in any study that goes beyond 
simple counting of heads. Evaluators should be sure that the project iias access to 
appropriate and adequate computing resources for establishing files and coding 
information— especially the right software and personnel who know how to use it. Most 
.school districts ha\-e computer sen'ices staff who can be enlisted to help. 
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In addition to this handbook, there are many written resources that can aid program 
managers in learning more about school- and community-based prevention program 
evaluation. The appendix of this handbook lists several relevant evaluation guides as well as 
information on compilations of survey instruments that address drug use. It also includes a 
list of the U.S. Depiirtment of Education Regional Centers for Drug-Free Schools and 
Communities, which pro\'ide assistance in evaluating drug education programs. 

Addressing Prcctical Problems in Planning and Implementing Evaluations 

Some problems arise so often in conducting evaluations that they are noted here. 
Several of these problems icre discussed in greater detail later in the handbook. 

There may be governmental or institutional research regulations to meet 
in drawing up an evaluation plan. The restrictions that apply when Federal education 
funds are used to ask questions of children are addressed in the next section of the 
handbook. In addition, states have ailes regarding research with human subjects, especially 
when programs ask questions about behavior. These rules may deal with confidentiality or 
parental consent. Evaluators must ensure that all such requirements are addressed when the 
evaluation is being planned. State offices that coordinate drug education programs should 
be able to inform schools and agencies about any state legal requirements. 

There may be difficulty in determining appropriate groups with which to 
compare program participants in order to study program outcomes or impact. If 
there is no group in the school, district, or agency that does not recei\'e the service, it may 
be difficult to find a group with which to compare participants. Although establishing a 
comparison group to study outcomes may be problematic, comparisons such groups are 
one way of demonstrating program impact. Possible solutions to this problem are discussed 
later in the handbook. 

Existing data sources may not be of sufficiently high quality to yield 
meaningful evaluation results or may not be kept in a usable format. For example, 
school records on disciplinary actions may be incomplete or lacking in detail. Knowing the 
limitations of available data early on will allow e\'aluation planners to gauge the amount of 
effon and time needed to collect additional information. Knowing in advance that available 
data are not sufficient can enable project staff to seek additional financial support and 
administrative or other approval for further data collection. 
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Evaluation tasks 
iviU invariably take more 
time than is originaUy 
anticipated. Establish a realis- 
tic schedule, and a budget that 
includes a little more resources 
than are initially anticipated to 
be necessar\*. It is always easier 
to reallocate unneeded resources 
than to find additional resources 
to fill an underestimated need. 

For example, procedures for 

obtaining written informed 

parental consent for student data 

collection can take a great deal 

of time. Or questionnaire 

responses may be incomplete 

and additional follow-up may 

be necessar\*. Evaluators 

should not expect everything to 

run smoothly. 

Not every data collection strategy will be implemented as plannedy so 
evaluators should prepare for contingencies. In other words, evaluation planners 
should not put all of their information "eggs" in one data collection "basket." It is useful to 
begin an evaluation with multiple data collection strategies or alternatives in mind. For 
example, a well-written sun'ey can still have a low response rate because of high 
absenteeism on the da\ s set aside for data collection. Or a comparison group can become 
•contaminated** by inadvertent exposure to the program, reducing the group s value. 
Program participants may drop out of school subsequent to the program, making it difficult 
to find them for a follow-up sun-ey. Hven if substitute approaches are more limited, they 
will be better than not completing the e\-aluation. 
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Obtaining Technical Assisiance From Outside the Project 



Program adminisimiors may want to consult in-house (district or agency) evaluation 
specialists, staff of other programs who have conducted evaluations, or groups established to 
provide technical assistance (such as the U.S. Department of Education Regional Centers for 
Drug-Free Schools and Communities) on specific elements of assessment. Or the staff may 
want to commission an outside individual or group to conduct the evaluation. In selecting 
individuals or groups to provide technical assistance, it is important to assess their likely 
contribution before making a choice. Among the elements to consider are the following: 

♦ The individual or group should have sptrcific background and experience in 
conducting evaluations of school- and community-based alcohol and other 
drug prevention programs, 

♦ The individual or group should be able to offer assistance with a variety of 
quantitative and qualitati\'e evaluation techniques in order to allow flexibility 
in evaluation planning (unless, of course, the program seeks consultation in 
some specific area such as statistical analysis). 

♦ The individual or group should be sensitive to the program goals, and to 
values and attitudes of the school or community^ in which the evaluation will 
be conducted. 

For anv r.\ pe of e\*aluation, a speciali.st can help pro|ect staff develop the evaluation 
plan by asking appropriate i^uestions, providing examples of comparable evaluatk)ns 
conducted elsewhere, reviewing drafts of work plans, and helping to make sure that the data 
will yield the t\'pe of information scuight. Fvaluatk)n design is a critical step for which .staff 
should seek guidance if expertise is not available in hou.se. No matter who is consulted. 
howe\'er. those in\-ol\*ed with the program must agree to the questions and procedures. 
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Consultation and technical as.si.stance resources include the following: 



# The school district evaluation or program assessment office: 

♦ Local universities (talk with people in the depanments of education, 
sociology, public health, etc.): 

^ State agencies charged with implementing Federal or state daig and 
alcohol education programs: and 

^ The U.S. Depanment of Education Regional Centers for Drug-Free 
Schools and Communities i names and addresses are included in 
the appendix). 



Chapter 1 of the handbook explores evaluation design in greater detail. It begins by 
introducing a fictitious school district searching for a new daig and alcohol prevention 
program. The Mor\' provides the basis for examining each major step in ex aluation. The 
chapter concludes with a discussion of evaluation findings and how they can be presented. 
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2. Steps in Designing Program Evaluations 



The Context for an Evaluation 

To set the stage for a discussion of the components of evaluation planning, we turn 
to a fictitious district, Wood County, which is about to adopt a new drug education program. 



Wood County School District Adopts a New Drug 
Use Prevention Program 

Parent leaders of Wood County School District have become 
increasingly concerned about drug and alcohol problems among 
students at Wood Regional High School. In the past year, four 
students have been caught with drugs at the high school and, in the 
past month, a party attended by more than 100 seniors was 
characterized by heavy drinking. Three students leaving the party 
had a near-fatal automobile crash. The parents demand that the 
district change its approach tu drug and alcohol abuse prevention. 
In response, the school board asks the superintendent to take 
immediate action. 

Advocates of a variety of approaches are invited to make 
presentations before a committee of parents, administrators, and 
teachers convened by the superintendent. Each makes a persuasive 
argument for his or her program and cites evidence of its 
effectiveness. After much discussion, the group decides to 
recommend that the district adopt the "Way to Go" program: a 
combination of policies aimed at deterring the use or possession of 
drugs at school (calling for suspension and referral for mandatory 
substance abuse assessment, or expulsion); a curriculum for grades 
9-12 that incorporates drug prevention education into science, 
health, and physical education classes: parent education: and a 
system of voluntary referrals for drug and alcohol treatment. The 
recommendation is presented to the Wood County School Board 
for approval. 

The school board is impressed with the **Way to Go'' program 
but is skeptical about some of the program's effectiveness claims 
and concerned about its cost. Not only are initial expenses for 
program materials considerable, but there will be substantial 
expenses for in-service staff training. Board members want 
assurances that the program will really change school practices 
and instrucUon. Most important, they want to know whether it 
"works," whether it convinces students not to take drugs or drink, 
and they want to know sooner rather than later. Seeking quick 
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approval, the superintendent assures the board that the district will 
evaluate the program and report back within a year. The board 
approves the "Way to Go" program. Unfortunately, no additional 
funds are authorized to assess its effectiveness. ^ 

A week later the district officials responsible for 
administering prevention programs meet to figure out how they will 
respond to the board s request. They, too, want to find out if the 
program works, but they face a number of potentially conflicting 
pressures. They helped to pick the program from among others so 
they have already become at least partially convinced that it works. 
In addition, they played a key role in recommending the program to 
the bo^d. so they could well be embarrassed if the program proves 
to be ineffective. They are also concerned that teachers will resent 
the time spent on evaluation as intruding on instruction. And, 
finally, they have little in-house expertise and few resources to 
conduct an evaluation. 

Ms. Brown, the district drug program coordinator, has 
thought about the problems ahead of time and now offers her 
recommendations. Based upon a consideration of the statement in 

Chapter 1 of the handbook. "I need to know because I need to 

decide ^ as well as an assessment of time, in-house evaluation 

capabilities, and resources, she makes the following comments: 

First, the program we have selected has a large 
number of elements and we have few resources. As 
a result, we need to narrow the evaluation scope in 
some manner. I recommend that we focus the 
examination of program outcomes on the policy and 
curriculum components because they should affect 
the largest number of students. 

Second, before we study whether the program is 
effective, we need to know whether the elements of 
the program are in place. Not only has the school 
board asked for this information, but we had 
considerable trouble getting the teachers to adopt 
the previous curriculum and we never really knew 
whether they were using it properly. Furthermore, it 
won't make sense to focus on outcomes if we can't 
show that changes were the result of the "Way to Go** 
program. We need a plan for tracking the adoption 
of the program, especially the components whose 
outcomes we plan to study. 

Finally, to ensure that the teachers are with us on 
this, we need to enlist their advice in drawing up the 
design. We should ask them to help us identify the 
effects that will be measured and, perhaps, some of 
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the specific questions that should be asked of 
the students. 

I believe that adopting this approach should not 
cost a great deal of money but will help us to learn 
whether the program is being adopted and whether 
it is effective. It will also build our credibility with 
the school board. Even if we find out that some 
elements are not effective, we will be able to 
recommend appropriate steps that can be taken 
to revise the program. We will be seen as 
innovators, critically trying out new approaches. /7 

Needless to say, the recommendations are compelling and 
the others approve the plan in general terms. But they also decide 
to form an ongoing evaluation committee. In addition to two 
administrators, membership is expanded to include two teachers, 
two parents, and a student leader. The committee will review 
available evaluation approaches and methods. After the review, it 
will make iinal decisions on evaluation design and methods. 



Now it ih lime lo translate Ms. Brown s recommendations into a plan. This chapter 
explores the elements in designing an evaluation plan, referring back to Wood County to 
highlight key points. Issues discu.ssed in this section include how to refine the evaluation 
(questions: track program implementation: identic* appropriate groups with which to compare 
program participants and determine project outcomes: a.ssure that data collection instruments 
are appropriate, x alid. and reliable: and meet the need for confidentiality in obtaining 
information about individuals. The section also discusses data analysis and presentation of 
evaluation results. 
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Refining The Evaluation Questions 



As noted in Chapter 1. the first task in any evaluation is to identify and narrow the 
questions that will become the focus of the evaluation. 



Tne Wood County School Board helped to simplify the choice 
of evaluation questions by providing general guidance about what it 
wanted to know. First of all, it wanted to loiow whether the 'Way to 
Go' program was actually adopted. The answer to this question was 
important in itself, to allow the school board to determine whether 
funds were properly spent and services delivered as intended. Ms. 
Brown recommended narrowing the inquiry further, to focus on the 
program's disciplinary policy and curriculum. Tracking the installa- 
tion of these parts of the program could also help the staff to identify 
problems in implementation and to make corrections. And, as 
Ms. Brown pointed out, documenting the extent and nature of pro- 
gram implementation would also be an important precursor to 
studying program effects. The committee meets repeatedly over the 
next several weeks to decide on specific questions that will address 
both program operation and program effects. It, too, asks what de- 
cisions will need to be made as a result of the evaluation, and what 
information is most critical to msiking those decisions, keeping in 
mind their limited resources and time. 



This section explores some of the issues the committee will confront as it tries to decide 
exactly what questions to select. 

Documenting and Analyzing Program Installation ond Operations 

FA'aluation of the implementation of a program is a good example of process 
evaluation. The obiect is to understand how program plans and objectives are translated into 
action. Process evaluation may be undenaken in order to monitor project activities, ensure 
accountability (e.g.. demonstrate that planned services are delivered in a timely manner), 
improve .^Jerxice deliverv (e.g.. ideniitS' obstacles and adjust activities), or set the stage for 
assessment of project outcomes. Some audiences may be most interested in accountability, 
but others may be more interested in impro\ ing service delive^\^ 

Monitoring the implementation and operation of a program usually invoK es 
identifying and tracking specific program acti\ ities or objectives. For example, if a first step 



in implementing a program is teacher training, the evaluation could examine the 
implementation of the training component. Here are a number of specific evaluation 
questions that could be asked about implementation of teacher training: 



1 . Did the activity (or procedure J aimed at hanging about program 
implementation (in this case, the training) occur as envisioned'' If noL what 
harriers or obstacles prevented pans or all of the activity from being executed? 

To document that an activir\' is conducted as planned, the evaluators need a means 
of monitoring the activity. Monitoring of teacher training, for example, might entail no more 
than checking the sign-in sheets to find out how many staff attend, whether the staff who 
attend are those who are supposed to implement the program, and whether they stay for the 
entire training session. .\ more ambitious plan might involve exit questionnaires or enlisting 
an objective party (someone unconnected with the training) t(^ ohseive the training and to 
write an assessment of how well the content of the sessions reflects program objectives. 
These obser\'ations could be carried out by the drug program coordinator, a district 
administrator, or someone hired b\* them specifically for this purpose. 

2. Did the clients (or participants) find the activity usefui and did they plan to 
use what they had learned:^ 

Asking people to implement a curriculum or staging a training session to introduce a 
program is no guarantee that implementation will occur. To get a better fix on the likelihood 
of use. the evaluators could ask the participants about the (}uality and userulness of the 
information designed to aid implementation (such as a curriculum guide or trainer 
presentation). That information would allow the persons charged with fostering 
implementation to get immediate feedback on how well they are teaching or otherwise 
transmitting information and to make any necessar\* adjustments. / survey of training 
participants, for example, could provide a measure of the training s effectiveness— if the 
teachers say that they plan to use the information, the in-ser\1ce session may be deemed 
effecii\ e. at least in a limited manner. 

.1 Did the training result in the program operating as planned? 

This is clearly the most critical c|uestion about the implementation of a program, 
since it asks whether training approaches, curriculum materials, or other information or 
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techniques are being used by teachers in the classroom. It is both an implementation 
question (e.g., determining the result of initial training activities) and an intermediate 
question for a student outcome examination (because it must first be demonstrated that 
teachers implement the program before saident outcomes can be measured). There are 
many ways that program operation could be measured. All teachers (or some teachers) 
could be asked a few questions about what they are doing in their classrooms, such as how 
often they use certain curriculum materials, whether they have completed particular 
curriculum units, or how many class sessions have been spent teaching particular skills to 
students. An unobtrusive measure of program operation could be a review of teacher lesson 
plans submitted to the school or distria. To find out whether administrators are 
implementing new discipline policies, the district or agency could review disciplinary or 
treatment referral records. A more ambitious plan might entail classroom observations and 
interviews of teachers or administrators by district staff or an outside evaluator. 

The choice of approach will depend, in part,, on the reasons for conducting the 
assessment. If the district's goal is to document implemt tion (e.g., to show a funding 
source that the program was adopted), then a short quesuonnaire filled out by a sample of 
teachers (or a review of randomly seleaed lesson plans or disciplinary records) may be 
sufficient to demonstrate the extent of program operations. Based on the findings, the 
evaluator might conclude that "Eighty percent of the teachers who attended training report 
that they used the curriculum as prescribed" or that "Suspensions for alcohol offenses 
increased 60 percent after the introduction of the new disciplinary policy." If the district's 
goal is to gain high (or improve previous) rates of implementation, the inquiry' might not 

only want to determine whether most staff are carrying 
out the program at some level it might also pinpoint 
where and why the program is not being implemented. 
The barriers to implementation could then be addressed. 

If a further goal is to find out how program 
exposure is related to student behavior, ascertaining 
the degree of program implementation will require 
more detailed information. For example, the evaluators 
mav decide to ask students how much exposure to 
''Way to Go" components they received, and ask 
teachers how many class sessions they devoted to 
"Wav to Go" instruction. 



If a goal is to find out 
how program exposure 
is related to student 
behovior, ascertaining 
the degree of program 
implementation will require 
detailed information. 
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After careful discussion of alternatives, the Wood County 
evaluation committee decides to focus its examination of program 
implementation on finding out (a) whether the appropriate staff 
receive in-service training: (b) whether teachers who receive training 
appear likely to use the new program: (c) whether teachers do, in 
fact, implement the program: and (d) whether the tough, new 
disciplinary policy at Wood Regional High School is enforced. To 
answer these questions, the committee proposes — 

# A review of attendance at the first teacher training 
workshop to determine whether attendees were those 
persons "targeted** for the training (i.e., those who 
would be most likely to implement the program) and 
what percentage of attendees remained for the full 
five days. 

# An end-oj-training questionnaire for all attendees. The 
"Way to Go" developers have a standard participant 
assessment form that they use in workshops. The 
district has arranged to add a few items that ask 
about the likelihood that participants will use the 
program. The developers will provide the district 
with the results of those items at no additional cost. 
Because most of the items have been used 
repeatedly, district officials reason that no pilot test 
will be needed and most of the items are likely to 

be reliable. 

# A follow-up questionnaire of all health, science, and 
physical education teachers, regardless of whether 
they attended the training. The question laire will 
ask about implementation of various components of 
the program, especially the curriculum and referral 
procedures. This questionnaire will be developed by 
the drug program coordinator and reviewed by the 
committee and the "Way to Go" trainer. 

# A review of drug-related disciplinary actions taken by 
the district in the year prior to the introduction of 
the new program and in the current year. 
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Observing Behavioral Outcomes and Attributing Chonges to the Program 



Many schools and community agencies will consider process e\*aluations to be 
sufficient for program assessment. They view their primary^ responsibility* as delivering 
services in an appropriate and efficient manner. They use evaluation to determine whether 
those goals are being met and to improve service deli\-ery where needed. 

But other schools, districts, and community^ agencies (much like the Wood County 
School District) will want to find out whether their programs are effecti\-e for Clients, whether 
they "make a difference" for recipients of services and others in the community- . They will 
want to know whether a set of interventions changes student beha\'ior or other indicators 
(such as DWI arrest rates) in order to decide how to proceed with their program. Outcome 
and impact evaluations identify changes that have occurred and analyze the changes to 
determine whether they are attributable to the program, that is. whether the changes 
would have occurred without the program activities. 

Demonstrating that changes in behavior occur as a result of a prevention program is 
not always simple, because behaviors such as alcohol and other drug use are likely to 
change over time. Children in programs mature, social norms change, new drugs are 
introduced, others become scarce. These conditions and many others can affect rates of 
alcohol and drug use independent of the effects of any specific activity^ (also called a 
"treatment"). A survey of participants in a program before and after a treatment may show 
that drug use did not increase, but critics may charge that the use rate would have remained 
stable anyway as a result of an intensive local media campaign or because of stepped-up 



drug arrests that diminished the supply of drugs to 



the community. 




The first step in conducting an assessment of out- 
comes is to decide exactly which changes are most 



in behavior occur as a result 
of a prevention program is 



important to measure. In the field of alcohol and drug 



abuse prevention, there are several common areas of out- 
come measurement for individuals, including changes in 
knowledge about drugs, attitudes about drug use. the 




behaviors such as alcohol 



ability^ to refuse drugs, and. of course, drug use. In addi- 
tion, programs are sometimes assessed for their ability^ to 
delay the onset of drug use. as well as for improvements 
in skills or attitudes identified in research as influencing 
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or mediating drug use such as positive self- 
concept or self-esteem. Increasingly, evalu- 
ations are looking beyond the immediate 
effects on individual participants to assess 
the impact of programs on communities 
through such changes as reduced drug ar- 
rests, fewer emergency room incidents, less 
drug-related violence, or other indicators of 
"community health." 

Once the specific outcomes have 
been identified, evaluation planners must 
explore how the assessment will relate 
changes in those outcomes to a service 
or package of services. There are a num- 
ber of possible approaches, some of which 
are relatively simple and inexpensive, while 
others are ambitious and require consider- 
able resources and planning. Some approaches focus exclusively on participants, while 
others compare participants with similar persons or groups. 

Assessment of treatment group onty. There are a range of possible comparisons 
that examine only the treatment group. The most common method is to collect data from 
the participant group both before and after the treatment. If this design were to be used at 
Wood Regional High School, for example, the lOth-grade students might be asked questions 
about their knowledge and use of alcohol and other drugs at the beginning and at the end of 
the school year (before the program begins and at the end of the first year of the program). 
The responses would be compared for evidence of changes in knowledge and behavior. An 
even more limited inquiry- might focus on one pan of the program— for example, a two- 
week instructional/skill-building component on drugs in health education— with pretests and 
posttests about drug knowledge or social refusal skills before and after that unit. This 
method would help the teachers to know whether students learned the information 
and skills. 

One-group pretest' posttest approaches are relatively inexpensive and easy to 
administer, but their main drawback is that it may not be possible to attribute changes in 
outcomes to the treatment. Suppose that a group of lOth-gmde students shows the same 
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BESTCOPUVNLABU 



]^e\ of drug use at the beginning and end of the school year. Should the program 

adiMSSt^ors conclude that a program offered in the interim was unsuccessful? Probably 
not. The reason is that lOth-grade students are at an age when experimentation with alcohol 
and drugs is likely to occur. It is possible that drug use would have increased without an 
intervention. But without information about the levels of use that might have occurred in 
tbe absence of the program, it would be hard to know whether the program had any 
effect on behavior. 

One way to address this concern might be tc compare changes in a treatment group 
to changes in some generally available "standard" of change in knowledge or behavior. For 
example, state agencies may administer questionnaires to students to find out about the 
prevalence of alcohol and drug use. The performance of a treatment group of 10th graders 
could be compared to that of "typical" ICth-grade students in that state's prevalence survey. 
Adjustments would be needed because surveys are usually administered only once a year, so 
they do not yield information on average or typical amounts of short-term behavior change. 
A partial solution might be to measure the behavior of lOth-grade students in Year One and 
that of llth-grade students in Year Two, comparing those results with similar populations on 
the state questionnaire. Then the change between local 10th- and llth-grade drug use rates 
could be compared with the overall changes for students of comparable age in their state. 

Of course, if students in a single school or district are not typical" of 
students in the state, comparisons with that standard may be inappropriate. 
Treatment students may have higher initial alcohol use rates than average lOth-grade 
students statewide, so that even after an excellent program, their use rates may remain 
higher. They may also experience personal or community conditions that make it either 
easier or more difficuk to refuse dnigs. Or in schools with relatively high mobility, many of 
the Year Two llth-grade students may simply not be the same students who participated in 
the lOth-grade program in Year One. State test administration dates and local program 
implementation may not be sufficiently complementary to make this kind of comparison 
possible, or the state may not measure or report behavior for the appropriate grades. 
Local programs may emphasize topics or behaviors different from those indicated in 
statewide surveys. 

National trend data can also provide a point of comparison. Bearing in mind the 
issues just discussed, year-to-year changes in knowledge or behavior among participants 
might be compared to changes reported in such national surveys as Monitoring the Future, 
an annual survey of drug use among students in grades 8, 10, and 12, and young adults; or 
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the National Household Survey on Drug Abuse, a j 

survey for which the responses of persons aged 12-17 can A IHOre rigOFOUS WOy tO 

be identified as a group. The appendix contains additional determine the effeCtS of Q 

information about these and other studies. 

treatment IS to compare 

Assessment of treatment and comparison 

groups. A more rigorous way to determine the effects of the performance of thoSC 
a treatment is to compare the performance of those who rOCeiVC the treatment 

receive the treatment with similar persons who do not 

c ' .r. r. Th.'.r Similar persons who 

receive it. Such persons form a companson group, liieir ^ 
knowledge, attitudes, or behavior are measured over the do nOt rOCeive it. 

same interval as that of the participants. If "before and 
after'' information is collected from both participants and 

these selected nonpanicipants. and if the group that receives services has a lower rate of 
substance a^' >se than the nonpanicipants after but not before the treaunent, it can be said 
with greater assurance that the program contributed to the change. 

One of the best ways to ensure that participant and nonparticipant groups are 
comparable is to assign people randomly to the treatment group and the comparison 
(or ""contror) group. This procedure reduces the possibility that the treatment group is 
different from the comparison group in a manner that can affect program outcomes. This 
procedure is commonly used in testing the efficacy of new medicines, but it is hard to 
accomplish in education programs. In a school, denying a new or potentially more effective 
treatment to a group of students is frowned upon, and even if it v/ere not. the treatment and 
control students might interact, thereby contaminating the comparison group. The process 
can be adjusted, however, so that classes or school buildings or community centers are 
randomly assigned to treatment or control status. 

An alternative means to create comparison groups is to divide the potential 
participants into several groups and stagger the treatment, with some groups participating 
in the first offering of a program and the rest in subsequent offerings. This approach is 
particularly attractive when a program does not have the resources to provide the services 
to all likely panicipanL<^ at one time, or when all students are not required to receive the 
intervention. Those ii the first group become the treatment group, and participants in 
subsequent offerings provide the 'comparison'* group. This approach only allows for 
shoit-term comparisons between groups, however, because eventually everyone receives 
the treatment. 
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To ensure that staggered groups are comparable, background information (e.g., 
gender, race, age, school attendance rates, academic test scores, etc.) should be analyzed. 
The more similar the groups, the more likely that any post-treatment differences between the 
groups are the result of the program. Even if the groups are somewhat different, background 
information can sometimes be used in statistical analyses to adjust for the differences. Or 
specific individuals from each group can be artificially selected for comparisons by the 
evaluators. However, approaches that ''match" students artificially are risky, and require 
considerable knowledge of evaluation methods. A variant of this approach is to compare 
current 10th graders with 10th graders at the same school in the past year, if necessary 
historical data are available. 

Perhaps there is simply no way to compare persons or groups in the same 
population that do and do not receive the treatment. This situation might occur if staggered 
treatment is impossible or if all possible subjects must be served together. Even in these 
cases, comparison groups may still be found. One possible comparison group might be 
students with similar personal and community characteristics (perhaps students in a 
neighboring high school or district). Once again^ background information (including gender, 
race, age, school attendance, test scores, etc.) must be used to identify the similarities 
between die groups at the outset and may be used to aid in statistical analyses of findings. 

The greatest problem in creating such matched comparison groups is knowing just 
what variables ought to be included in the match. If a match misses critical 
characteristics, the groups cannot be said to be truly comparable. It is easy to think of 
reasons why participant and comparison groups could be different. One common reason is 
that participants are chosen for a program in a purposeful manner. For example, participants 
in an agency-sponsored program may have volunteered to participate in the program. 
Volunteers are probably more likely to be helped by the program than are people in the 
same community, even when they share the same age, race, sex, socioeconomic status, or 
educational ability. Or participants may have been selected to receive the program because 
their school or community had specific problems. Such participants would be more likely to 
be troubled than a population that is similar with respect to age, sex, race, etc. Students 
may also have been scheduled into an intervention because they were all taking the same set 
of classes before and after drug education. As a result, they might share academic 
characteristics that could affea their drug-related behavior. The selection of appropriate 
treatment and comparison groups is an area in which program officials may want to consult 
with evaluation specialists within or outside of the district or agency. 
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After weighing considerations of accountability, rigor, and 
cost, the Wood County evaluation committee decides to limit the 
inquiry about the curriculum to the 300 students in the 10th grade 
at Wood Regional High School. The committee feels that focusing on 
a single age group will be sufficient to meet the school board's 
requirements while keeping evaluation costs relatively low. Ms. 
Brown reasons that 10th graders would be a good choice because 
she believes that sufficient numbers of students have started to use 
drugs or alcohol for the evaluation to measure changes in use. 

The committee wants to measure the knowledge, attitudes, 
and behavior of iOth-grade students in the fall, before "Way to Go" 
instruction, and again in the late spring (after "Way to Go" 
instruction). The committee notes that because few students move 
out of the district between 10th and 1 1th grades, it may also be 
possible to track the lOth-grade students over time. Most of the 
10th graders will also participate in the statewide prevalence survey 
that will be administered to 1 Ith-graders the following year. The 
Wood County 1 Ith-grade results on the prevalence survey will allow 
the evaluators to measure longer-term program effects. 

The committee decides that only an evaluation with a 
comparison group will be sufficiently rigorous to identify program 
effects. Because £l11 Wood County students will be exposed to the 
"Way to Go" curriculum at the same time, neither random 
assignment nor staggered services are feasible designs. Several 
committee members know the staff at a neighboring high school that 
draws its students from roughly the same ethnic and income 
groups. Those members form a subcommittee to approach the other 
school district about participating in the evaluation, using its 10th 
grade students as a comparison group. 

At this point, then, the Wood County evaluation committee 
has identified three separate data collection efforts: 

^ A pretest/ posttest survey of teachers: 

^ A review of disciplinary records; and 

^ A pretest/posttest survey of participants and 
nonparticipants in "Way to Go" instruction, 

with the state prevalence survey as an additional posttest. The use 
of multiple methods increases the likelihood that useful information 
will be obtained. It also allows the evaluators to compare the results 
of the different inquiries. 
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Who Should Be Surveyed? 



If a drug education program is adopted district-wide and the distria has 5,000 
students, surveying all of those students to identify changes in knowledge or behavior could 
be quite expensive. The addition of a comparison group could inaease costs even further. 
Rather than polling everyone, it may be possible to selea a smaller number of individuals 
and to survey them with some assurance that the results will be applicable to distria or 
comparison students as a whole. Selecting that smaller number of individuals is called 
sampling. Here is an example from another field that helps to explain sampling. 

The owner of a grocery store finds an unopened box in the back room with a label 
that says "48 cans, 16 ounces each...." The rest of the label is obliterated. He opens the box 
and sees that it is filled with cans. He takes out one of the cans and reads the label: It is a 
l6-ounce can of tomatoes. How reasonable would it be to assume that the other 47 cans are 
also full of tomatoes? He is a skeptic (a good quality for an evaluator), so he pulls cut three 

additional cans from different parts of the box and finds tb.at 
they are all l6-ounce cans of tomatoes. His confidence that he 
has opened a box containing 48 l6-ounce cans of tonriatoes has 
increased considerably. 

The process used to arrive at this conclusion is 
sampling. It is a way to be reasonably convinced of xhe 
characteristics of a larger population based on locking at 
a subgroup of that population. Sampling can save a great deal 
of time and effort in conducting an evaluation while still 
providing accurate results that are representative of a larger 
population. Here are five factors to keep in mind when 
deciding how to create (or *'draw") a sample. The faciofs affect 
how many things/persons should be sampled. 



Sampling is a way 
to be reasonably 
convinced of the 
characteristics of 
a lorger population 
based on looking 
at a subgroup of 
that population. 



General Sampling Considerations 



L Prior Imowledgc of the condidon being obscrvecL (What if the 
gixKer's box had no label at all and had already been opened?) 

The less that is known about what/who is 
being studied, the higher the number needed 
in the sample. 

2 . The number of things/persons from which to sample. (What if 
there had been a "population" of only two cans in the box?) 

The larger the number of things/people in 
the overall population from which the sample 
is drawn, the higher the number that should 
be sampled. 

5. Thevariabilityof the things/persons to be sampled. (What 
should the grocer have done if the box had said "assorted** or if the 
second can had contained com?) 

The more variability likely to occur in the 
population being studied, the higher the number 
that should be sampled. If there is a great deal of 
variability, a laige sample may be required. 

4. The importance of the clecisions that will be made based on 
the information obtained. (What if the example had involved 
unpacking bottles of vaccine for a childrens* clinic rather 

than tomatoes?) 

The greater the degree of confidence in results 
that is required, the higher the number that should 
be sampled. 

5. The reliability of the measures being used. (What if the labels 
on the cans had been partially obscured?) 

The more reliable the measure being used, the 
smaller the necessary sp.apie c^ze. 



Sample size will depend on all of these factors, .\rriving at the correct sample size is a 
technical process. An evaluation specialist can help in drawing up sampling plans. 



The Wood County evaluation committee decides to survey 
only 10th graders to keep costs low. The committee also could have 
used sampling and expanded the number of grade levels in the 
evaluation. This possibility was considered, and rejected, because 
the conmiittee reasoned that the costs of hiring a sampling expert 
and the difficulty of polling only a small number of students at each 
grade level would be . greater than the costs of polling all 10th 
graders. Of course, each evaluation committee will have to reach its 
own conclusions. 



Protecting Humon Subjects and Maintaining Confidentiality 

Infoimed consent Collecting data from human subjects is a delicate matter, 
especially when data collection involves minors. Organizations that use Federal funds to 
collect information directly from children must obtain parental consent. Each situation will 
differ, however, with respect to state laws and the policies of the district or organization 
conducting the program. There will be differences in whose consent is sought depending on 
whether the evaluation is using extant records, interviewing minors, or sending a 
questionnaire to parents. 

Under all circumstances, it is important to ensure that written informed consent is 
obtained from the appropriate source. Informed consent ensures that a person is aware of 
(informed oO what is being collected and gives his or her pennission to be part of the data 
collection (gives consent). Notification must be given in a language that he or she 
understands. Additional information on informed consent is available from district or state 
agencies. Local school program officials should become familiar with the consent policies of 
their districts. 

Evaluations conducted by district or agency officials that use students' records— 
cumulative or attendance records, for example— usually do not require the whtten consent of 
parents. Nonetheless, it must be established that there was informed consent for placing the 
information in the files and that the evaluator has complied with all agency or district 
policies for obtaining and using the data. 
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Anonymity and confidentiality. In examining the effects of programs aimed at 
preventing substance abuse, it is often necessary to ask for very personal information. That 
information is often requested primarily to understand the impact of a program and only 
secondarily (if at all) to assist the individuals being surveyed. Individuals are likely to be 
concerned about re\'ealing information that could prove emi:>arrassing or place them at legal 
risk. At the same time, there may be little individual benefit to providing sensitive 
information. It is necessary to reach a balance bet^'een the risks to these research subjeas 
and the benefits derived from the evaluation. 

What risks to subjects can arise in evaluating a program? First, there is the risk that 
damaging information about an individual may become known to project staff or even made 
public. There is the risk that information that was supposed to be held in confidence may 
be used in v\ ays that hurt individuals or groups of respondents. There is also the risk that 
some respondents may react poorly to questioning on for other reasons, the evaluation itself 
causes harm or may even undo project objectives. These risks are small but real, and 
evaluators should take steps to address them during the design phase. 

The best way to protect evaluation subjects is to ensure them anonymity. That is 
the case v^ hen the evaluators obtain no identifying information during data collection or 
expunge all ideniifv'ing information from any records or other materials they acquire. This 
approach has the added benefit of decreasing the likelihood that respondents will gi\'e 
inaccurate answers because they think the information ihcy are i^roviding could be traced to 
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them. However, ensuring anonymity prevents the evaluator from linkmg different sources 
of evaluation information (e.g., linking questionnaires administered to students before 
and after an intervention, or linking student responses with parent questionnaires and 
student records). 

From the evaluator's standpoint it is better to promise confidentiality. This means 
that each respondent is assigned a number that is used in data collection. A very small 
number of persons have access to the list that links respondent names and numbers, and that 
list is destroyed when the evaluation is completed. It may be destroyed as soon as data 
collection is completed if no individual follow-up is planned. The number does not need to 
be included on data collection instruments but it must be linked by the evaluator to any 
numbers that do appear on questionnaires or other instruments. In this way, the respondent 
is ensured that his or her identity will be known to only a very few persons. This approach 
provides less individual protection than does anonymity but it gives the evaluator the 
important ability to link data. 



Recog.nlzing the sensitive nature of the inquiry, the Wood 
County evaluation committee decides to send a letter to the parents 
of all lOth-grade students at the start of the school year. The letter 
win describe the new drug prevention progrsim, the evaluation 
design, and the range of persons and groups Involved in the 
evaluation. It will ask parents to indicate whether they consent to 
their children's participation in the evaluation. Because the 
evaluation committee plans to review the district's records on 
suspensions, expulsions, and referrals, as well as to administer 
questionnaires that ask for sensitive information, the committee 
decides to limit, as much as possible, the nimiber of persons who 
will conduct these data collections. Each person who will have 
access to confidential information will sign a form indicating that he 
or she will maintain the confidentiality of the records and will 
disclose no information on any individual. Because the evaluation 
committee wishes to link student questiormaires and teacher data, 
questionnaires will promise confidentiality to respondents, 
not anonymity. 



The rest of tills chapter outlines key data collection, analysis, and reporting aaivities. The 
Wood County approach to these aaivities will be discussed in Chapter 3- 
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Data Collection Methods and Instruments 



With the research questions identified, the 



design and sampling issues considered, and the means When Confidentiality js 

to ensure informed consent and confidentiality outlined . , 

the discussion turns to the methods for collecting data. ' ^«'f-°'''"»"«tered, 

Among the data collection options available for carrying ^'^^C" questionnaires Ore 

out an evaluation-reviewing existing records, observ- probably D Qood meonS 

ing services, conducting open-ended interviews, and 

administering written questionnaires-written question- ^okdmq information 

naires are one of the most common and least expensive relatively inexpensively. 

methods to collect information. At their best, question- 

naires provide an efficient, convenient means to 

produce systematic, measurable information about how a program is operanng or affecting 
clients. When confidentiality is assured, self-administered, written questionnaires are 
probably a good means of collecting information relatively inexpensively 

Selecting Questionnaires 

Most school .systems and community agencies do not have the resources to develop 
new, perfectly tailored questionnaires. Instead, evaluators often select all or parts of existing 
questionnaires that have already been tested, used elsewhere, and found to collect accurate 
.nformation.2 T^e advantages of using such instruments are considerable. Resources need 
not be spent designing new forms, and existing questionnaires often have procedures 
.ncluded for recording the information so that it can be collated or aggregated. As already 
noted, using questionnaires employed in national or regional surveys can sometimes also 
allow local evaluators to compare their results with national or regional results. 

Of course, the use of pre-existing instruments also has disadvantages It can be a 
long, frustrating process to find appropriate questionnaires. When items are drawn from 
several different instruments, they may not fit together easily. Or it mav be necessary to 
change parts of questionnaires to adapt them to local needs. Merging or changing items 
may. however, alter or reduce the evaluators level of confidence that those items are 
collecting accuse information. To begin the .search for questionnaires, several compendia 
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of alcohol and drug questionnaires are identified in the appendix. They address different 
topics and offer different emphases (including measures of drug-related behavior and habits, 
level of knowledge about drugs, attitudes about drug use, and related topics). When 
examining drug-related behavior, it is helpful to use existing questionnaires if at all possible. 
In addition to saving resources and ensuring a reasonable degree of accuracy, they offer 
comparability between populations and over time. Designing an entirely new questionnaire 
is best suited to testing a unique component of a program. 



Designiiig Questionnoires 

Whether choosing a pre-existing questionnaire, collating one that is partly or wholly 
adapted from others, or creating a new one, here are some tips to ensure that the produa is 
of high quality. Whatever approach is chosen, the evaluator should ensure that: 

♦ The items in the questiomiaire reflect the program's specific aims. For 

example, what changes in participant behavior might be expected to occur 
as a consequence of the prograra^ What are the best indicators that teachers 
are implementing a curriculum? The questions should measure what the 
program is designed to achieve. 

♦ The questions, language, and reading level are appropriate to the 
respondents. The flow of the questionnaire and the ease of responding 
should be assessed. For example, the questions should not ask young 
children about issues they are unlikely to comprehend or about behavioral 
expectations well beyond their stage of development. 

♦ Wording biases have been eliminated* If the wording of questions leads 
respondents to guess the desired ariswer, nothing will be learned from 

the questionnaire. 

♦ Questions are direct and focused, not indirect or open ended. Yes/no or 
simple 5-point scales (such as "strongly agree" to "strongly disagree" with a 
particular statement) should be used when possible. 

♦ The response format matches the question format A question that asks 
"How many times in the last month..." should have response choices tailored 
to that question. 

♦ Coding requirements are incorporated into the instrument The 

person who will be responsible for coding should review the instrument to 
make sure it will be easy to code and aggregate the information. 
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♦ Items from widely used tastniments are not changed unless there is a 
good reason. In addition to issues of accuracy, comparability of findings 
with other evaluations or national trends could turn out to be of critical 
importance. Each item change should be weighed carefully. 

♦ The instrument Is sensitive to response burden--the number of minutes 
it will take a respondent to fill it out— and to the burden on evaluator time. 
A long, poorly thought out questionnaire will waste respondent and 

staff time. 



Administering Questionnaires 

Before a questionnaire is administered as pan of an assessment, it is important to try 
the items with a small number of people: this is commonly called z pilot test A pilot test 
helps to ensure that the questions are understandable and answerable and :hat the time 
needed to complete the questionnaire is not excessive. The pilot test should be conducted, 
under similar conditions, by the persons who will administer the actual questionnaire. 
Respondents should be similar to the actual respondents but not persons likely to participate 
in the tnie questionnaire administration. The pilot test could be administered, for example, 
to a group of students in a neighboring school or district. In most cases, a dozen or fewer 
pilot test respondents will be enough to know how well the questionnaire works. When the 
administration is over, the respondents should be asked for their opinions of the instrument. 
After the pilot test, some redrafting of the questionnaire may be necessary. 

With respect to the setting for actual administration, questionnaires administered in 
person or over the telephone tend to produce higher response rates than those administered 
by mail, when evaluators are dependent on subjects to mail in their responses. But when a 
large number of people are being polled, and they cannot be brought together easily, a mail 
questionnaire may be more cost effective. And some evaluators have argued that telephone 
interviews may not be as effective a method when sensitive information io being sought and 
the respondent knows the identity' of the interviewer or the group conducting the survey. 

There are a number of other points to keep in mind as well. First, time and 
resources must be available to follow up nonrespondents to ensure a high response rate. 
Computer service staff should be enlisted to help design forms. The evaluators should 
identify as many costs as can be anticipated at the outset, including copying, collating, 
postage, etc., and try to keep the questionnaire as simple as po.ssible, given the resources. 
As noted previously, pre-coded identification numbers help assure confidentiality. For a 



written questionnaire administered in person, the evaluator might consider giving each 
respondent an envelope for the completed survey and providing a box where sealed 
envelopes can be placed. If at all possible, someone other than a familiar classroom teacher 
or program leader should administer the survey to participants (and refrain from watching 
respondents as they fill it out). This procedure will help to eliminate inaccurate responses. 

Using Doto from Records 

Data gathered from ongoing recordkeeping can be a very useful and inexpensive 
source of information for program evaluation. Examples include school records of 
attendance, grades, referrals, and disciplinary actions. Community agencies may have access 
to police or hospital emergency room records. Such records can be especially valuable 
when combined with the results of new data coUeaion. Student self-reports on achievement 
can be compared with school cumulative records, for example, to establish whether the 
direction of change is the same for both sources. Sometimes new recordkeeping is 
established by the evaluators solely for the purposes of the assessment. 

Programs with intensive direct services to a subset of students or a group of clients 
are likely to employ case management or other individual data files. When standardized 
forms are used to record information about individuals, they can also yield information to 
monitor and evaluate program operations. For example, a community-based counseling 
program with a file on each child that includes items such as gender, age, parents* education, 
family income, and services rendered could be used to retrieve information on how much of 
a given service the program provided and how many people were served, or to 2issess 
whether the goal of serving children from low-income families was accomplished. The 
records also might yield information such as the average amount of service provided to each 
client or the average caseload for each staff member. Case records of this type are more 
likely to be available in community agencies than in schools. 

Since records data are usually collected for purposes other than evaluation, their 
accuracy must be determined. Also, recordkeeping procedures may change over time, 
producing inconsistencies in the data. It is important for evaluators to assess and understand 
the limits of records data, but with these cautions in mind, they should take advantage of 
opportunities to use records reviews in combination with new data collections whenever 
possible. If multiple data sources show decreased drug use over time, for example, 
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confidence in the results may be strengthened. It is critical, however, to understand the 
policies and procedures that have influenced the quality of the data being used. 

Many agencies and school districts require a formal application process for obtaining 
information from individual records. When seeking access to these data, the evaluator must 
be very specific in describing the information needed and how it will be used. He or she 
should be prepared to work collaboratively with the district or agency to obtain the data in 
the least intrusive manner and to protect confidential material. 

Ensuring That Evoluations Yield Voild and Reliable Findsigs 

Whatever methods are used, the data collected must meet two conditions to be 
considered accurate: they must be valid and reliable. In examining sensitive issues such as 
drug or alcohol use among youth, designing data collection instruments and methods that 
yield valid, reliable findings is a very serious concern. Respondents may be tempted to 
answer questions in ways that they ttiink are expected of them or that do not place them in 
jeopardy. Evaluators will want to take steps to ensure that they have obtained the most 
accurate (i.e., valid and reliable ) responses they can get. 

A data coUection item (such as a question on a questionnaire) 
is valid to the degree that it actually measures what it claims 
to measure. 

For example, a pencil and paper questiormaire on drug use is administered to a court- 
referred group of drug users obtaining counseling. Less than 5 percent of the respondents 
indicate any drug use in the past year. The validity of this instrument would be highly 
questionable. Or a middle school invites a police officer to a health education class to 
present information that will discourage students from using marijuana. At the end of the 
session, the officer asks the participants, "Will you smoke marijuana?" It is likely that just 
about everyone will say "no." If everyone says "no" in this public manner, little if anything 
has been learned about these children's attitudes toward future marijuana use. There is no 
validity to this procedure. A better procedure might be to wait a few weeks and then 
administer a short, confidential questionnaire asking about the possibility of future use. 
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A measure is reliable to the degree that its meaning is stable. A 
reliable item or set of items on a questionnaire would lead to 
similar responses by the same respondent (in an unchanging 
siUiation) each time the item was asked. 

Reliability is an assurance that the instrument or measure is consistent. One of the simplest 
tests of reliability is whether the same questionnaire, administered to the same person twice 
in a short period of time, yields similar responses. If this does not happen, the questionnaire 
probably contains unreliable items. Consistent responses suggest reliability, and consistent 
responses to different items that seek to measure the same knowledge or behavior provide 
greater confidence that the questionnaire is reliable. 

Can a measure be reliable but invalid? ^'es, because reliable but invalid responses 
can be obtained consistently if a data collection instrument or procedure is poor. An 
evaluator administers an instrument that asks about current drug use to the same set of 
students twice, a month apart. The students give the same answers both times, so the 
questionnaire appears to be reliable. But the evaluator asks the students to sign their names 
to the questionnaires, so the students assume that if they reveal any drug use they will be 
disciplined. Hence, reliability is achieved because drug use rates at both administrations are 
consistently low. So the measure is very reliable, but still invalid. 

A measure must be both valid and reliable to be useful Establishing the 
validity and reliability of data collection methods and items is a technical area in which 
school staff may wish to obtain outside assistance. The use of validated existing 
questionnaires is a good way to minimize validity and reliability problems, provided they are 
used in a manner similar to the way in which they were used when their reliability and 
validity were established. 

Interpreting and Reporting Evaluation Findings 

Drafting an Analysis Plan 

An analysis plan links the evaluation questions with the data collection and spells 
out the analyses that will be conducted when data become available. It should be written in 
conjunction with the drafting of data collection instalments rather than afterv.'ard. If the 
analysis plan is written too late, key questions may not be asked or may be phrased in 
unfortunate ways, or critical information from other sources may be overlooked. 
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As questionnaire items are prepared or data 
entry' forms developed, the analysis plan links these 
data collection elements to the analyses that will be 
conduaed. For example, the plan might say: "Because 
the distria wants to know whether the program is 
effective for lOth-grade students, the analysis will focus 
on whether attending a greater number of class sessions 
is associated with reduced alcohol use. Items X and Y 
on the student questionnaire will measure alcohol use 



An analysis plan links the 
evaluation questions with 
the data collection and 



spells out the analyses that 



will be conducted when 
data become available. 



(before and after the intervention), while teacher 
attendance logs will be used to determine the number 

of drug prevention sessions a student attended." The analysis plan can also identify the 
specific range of answers that will be solicit*, e.g., for a question about alcohol use the 
range might be "never," "once or twice in lifetime/' and "a few times a year," if very little use 
is expeaed). If items are -specified in detail, and analyses are linked before instruments are 
completed, it will avoid disappointment later. (*Tf only we had asked about monthly or daily 
alcohol use, we could have learned more about actual use rates and related them to 
program exposure.") 

Development of the analysis plan is a point at which the help of an evaluation 
specialist can prove useful. A specialist from within or outside the district or agency can 
review the plan and the data collection items, to make sure that the evaluation questions are 
capable of being measured, that the data collection instruments will yield data that can be 
used to answer the questions, and that the planned analyses will use those data properly. 
He or she can raise questions about whether a "yes" answer to teacher questionnaire item 8 
and a "3" answer to student questionnaire item 10 will allow the evaluation to conclude that 
students were more likely to refrain from using alcohol when teachers emphasized the 
refusal skills approach. The specialist can help the evaluators to develop ways of 
characterizing programmatic differences across staff or institutions. Showing these differences 
may be critical to understanding both program implementation and participant outcomes. 
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Airalyzing Evuluotion Findings 

Once data colleaion is completed, the process of data analysis begins and the 
effects of the program emerge. A basic tool for analyzing data is descriptive analysis. 
Descriptive analysis may be as simple as summing or averaging results: How many 10th- 
grade students report that they know someone who uses marijuana? What percentage of 
10th graders report drinking alcohol twice a week or more? What was the mean score of 
participating students on the drug knowledge items? What was the mean score of the 
comparison group? 

In a process e\-aluation. statistics will likely be relatively straightforv^-ard: for 
example, the number of persons sensed with this program in place (possibly compared with 
the number of persons ser\'ed before this program was in place), the number of instructional 
hours, the number of counselor contact hours or individual sessions, the number of staff 
trained, etc. Descripti\'e information should be presented objectively, in quantitative terms 
where possible. 

Descriptive analysis also extends to characterizing the relationships between 
different measurable aspects of the program. By examining relationships using statistical 
tools as well as common sense, it may be possible to show, for example, whether attending 
in-ser\Mce training sessions is associated with a greater likelihood that teachers adopt a new 
curriculum, or whether teachers who believe that the school has a serious drug problem 
refer n^ore students for treatment. Carrying out this t\'pe of descriptive analysis requires 
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crossiabulations, correlations, and other statistical techniques designed to depict relationships 
betvv'een variables. These techniques cannot establish causality, however. 

Analysis can also be inferential in nature. In an outcome or impact evaluation, 
there are a variety of questions about effects of a treatment on an observed result. A simple 
inferential analysis might seek to determine whether observed differences in outcomes 
between treatment and comparison groups are statistically significant, or whether it is likely 
that they could occur by chance. A more complex question might be whether differences 
betw^een treatment and comparison groups are significant vvhen the background or previous 
drug education of the two groups are taken into account (i.e., "held constant*'). The help of a 
statistician or evaluation specialist may be needed to carr\' out this type of analysis. 

Evaluators are also called upon lo interpret data. For example, even if a 
statistically significant difference between groups is obser\'ed. is that difference meaningful in 
a practical sense? What does it tell program personnel that can help them to improve the 
program? Let's say that among lOth-grade students whose health teachers indicated that they 
implemented a new curriculum in its entirety-, 3 percent were suspended or expelled during 
the evaluation year. By contrast, 5 percent of the 10th graders who were not taught these 
skills were suspended during the same time period. While a statistical test shows that the 
difference between the suspension rates of the two groups is statistically significant (i.e., the 
difference is not likely to ha\'e occurred randomly), is that difference meaningful in a 
practical sense? Is the difference sufficient to continue the program in its present form, or 
should the staff make changes? Should every high school student in the district receive the 
new program if substantial additional teacher training costs will be incurred? 

A final word on data analysis: Evaluators should not be discouraged by findings 
indicating that a program demonstrated few effects on participant behavior. It is difficult 
to evaluate a program that invoK'es such a complex, 
sensitive issue as drug use. It is particularlv hard because 

u u , u r 1 n r f Progroms that are new, 

behavioral changes may occur for only small percentages or ^ 

participants or changes may be difficult to measure. OF that are recently 

Programs that are new. or that are recently adopted, may adopted moy not be 
not be sufficiently de\ eloped or implemented to show 

behavioral effects. One way to a\-()id disappointment is to Sufficiently developed 

make sure that the evaluation questions address changes or implemented tO shoW 

that could reasonably have been expected to occur in the l l • i u ^ 

behavioral ettects. 

time frame under examination. 



Reporting Evaluation Rndings 



Once the findings are in, evaluators need to develop strategies to report the results. 
The form of the report depends on the audience. There may be several audiences for a 
program evaluation, each requiring different information or a customized presentation 
format. An in-house evaluation conducted to improve a program's implementation will lead 
to a report that is very different from that required in an "accountability" evaluation 
conducted for a funding source or the school board. A report intended for media use will be 
different still; it may, in faa, be a press release. 

While most evaluations include a detailed written report, this approach is not 
always necessary or appropriate. Verbal presentations with supporting tables, graphs, and 
charts— or case studies and targeted qualitative results— may be enough for some needs. But 
most evaluations will call for a report summarizing the goals and history of the program, 
methods of evaluation, findings, interpretations, conclusions, and recommendations. 

It is important to review the results with colleagues and program staff before 
completing an evaluation report. This review can be accomplished by circulating an interim 
or draft report and holding a meeting to discuss it together. The evaluator will gain 
additional perspeaive on the meaning of the data from the reviewers before he or she writes 
the final draft. For example, colleagues can discuss and interpret puzzling findings. Perhaps 
participants reported increased use of drugs after the treatment but less intention to use 
dnjgs in the future. Discussions with staff can bring out new perspectives on the meaning of 
these findings, perspectives that can be included in the final document. 

It is also a good idea to brief important political actors before a report is released 
publicly. The briefing gives district or agency officials or other policymakers some time to 
digest the findings and to think about the policy implications. It may also provide them with 
an opportunity to prepare a response if they so desire. Through this process, evaluators will 
also learn what appear to be the most important findings from the perspective of the groups 
that will use them. 

The final report can be written as a short summary document with a technical 
appendix. Most busy audiences look for brevity and clarity. In summarizing findings, the 
evaluators should not be afraid of appearing too elementary. Those who want more 
statistical or other detail can find it in the back-up technical report. Evaluators or program 
staff may also be called upon to make oral presentations before various groups. Speakers 
should prepare a set of perhaps half a dozen simple summary graphics (charts and tables) on 
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the most important findings to show on an overhead projector and to distribute. They can 
also prepare copies of a summary sheet of results to give to the audience. 



Whether oral or written, the report should begin by pointing out why the 
evaluation was conducted and asked the questions it did. It should state the 
purposes of the program and how it was developed or selected for the school or 
agency. The report should indicate what the board, staff, or others wanted to learn 
from the evaluation and why, and explain the methods and the procedures 
undertaken to collect and analyze data. The evaluator should share the highlights of 
the results and describe what the findings imply for program maintenance, 
expansion, redirection, funding, etc., as appropriate. The report may also advance 
recommendations for future steps, short- and longer-term actions that can be taken 
to improve the program further. 



It is crucial to report evaluation findings objectively. Most evaluations have both 
positive and negative findings as well as findings about which the evaluator is less than fully 
certain. Most studies have methodological and other limitations that constrain the ability of 
the evaluators to reach definitive conclusions. Evaluations also encounter external events 
that impede the data collection or analysis. These circumstances need to be discussed fully 
and honestly, so that audiences can judge the degree of confidence to place in the results. 
A good scientist works hard to discredit his or her own findings. 

Finally, the report should not portray the results as the final word on the program 
but should present the evaluation results as part of a cumulative and evolving process. 
Evaluation data represent one kind of input into decisionmaking, but there are other sources 
of information as well. And the evaluation is ultimately a device for program staff to make 
adjustments, to improve their effectiveness. The change process will probably continue. 

Now that the major parts of an evaluation have been identified, the experience of 
the Wood County School District is used to illustrate how to conduct data collection 
and analysis. 
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Implementing an Evaluation Design 



Chapter 2 presented the overall design for the Wood County evaluation. This 
chapter provides a detailed discussion of the data gathering and analysis activities carried out 
in Wood County. These activities are typical in canying out most evaluations of program 
implementation or outcomes. 

Establishing That Implementation Took Place 

In Wood County, the first order of business is to determine whether the new 
curriculum has been implemented. The evaluation committee plans to find out: 

♦ Whether staff received training; 

♦ Whether teachers who received training appeared likely to use the 
new program; 

♦ Whether teachers who received training implemented the program; and 

♦ Whether the new disciplinary policy was enforced. 

The evaluation committee wants to obtain this information using procedures that are easy to 
conduct, reflect good evaluation practice, and are as unobtrusive as possible. They want to 
use data collection instruments that yield valid and reliable data. As a result, they review 
published evaluations and evaluation guides, seeking to determine how other 
implementation assessments have been conducted. 

The committee soon discovers, however, that formal guides for studying the 
implementation of drug education programs are scarce. The few a\'ailable guides prove 
inappropriate to Wood Count\', because the "Way to Go*' program uses a teacher training and 
classroom approach different from that of other programs whose implementation has been 
systematically studied. 

The committee findb that questionnaires or other data collection instalments used in 
process evaluations can rarely be applied across programs. Unlike the measure .nent of 
program outcomes, the measurement of program implementation tends to be developed 
entirely "from scratch." or by cobbling together new components with some items drawn 
from other evaluations. \X'ith the help of Ms. Thrown, however, the committee draws up the 
following implementation data collection strategy. 
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Implementotion Question #1: Did Staff Receive Training? 



To establish which staff received training, the evaluation committee decides to 
record teacher attendance at the week-long training session held during August. During 
each day, teachers are asked to sign their names on a preprinted sheet. Since the "Way to 
Go" training is taught by several different instructors, Ms. Brown makes sure that each 
instructor is aware of the need to pass around the sheet once a day. Preprinting the list of 
names in alphabetical order will simplify the entry of results into the computer. A member 
of the evaluation committee collects the forms from the training instructors at the end of each 
day of training. At the end of the week, the evaluators are able to tell which teachers 
attended the full week and which missed 1 or more days. 



-Way to Go" StaiOf Training 
Attendance Sheet: Session 1 8/15/92 



Please sign your name in the appropriate space. 



Amy Adams 
Perry Black 
Elizabeth Edwards 
Eve Goodman 
Krist>' Jordan 
Paul Kee 

Patrick McCarthy 
Louise Martino 
William Moore 
Melissa Nelson 
Sylvia Rodriguez 
Ray Sanchez 
Eugene Simpson 
Michael Wood 



\ 




54 



implementation Question #2: Did StoK Appear Likely to implement 
^aytoGo"? 



To determine the immediate outcomes of the training for participants, the evaluation 
committee decides to administer a short questionnaire to participants. The initial plan is to 
rely on the "Way to Go" developers' training participant questionnaire, which the developers 
use to assess the training sessions. It asks participants about the usefulness and qualii\^ of 
the training they receive from each instructor, using a 5-point scale. The committee learns 
that the developers pass out multiple forms to each participant on the first day of training, 
and participants are asked to complete one form, anonymously, each day. A box for 
completed forms is posted at the back of the room. 



The commdttee considers using this questionnaire: 



Developere- "Way to Go" Dally Evaluation Foni. 
Please ...catewh*„youa^ord.sa^««a,eac.ofa,efo«o.„gsta— 
by placing an "JC In the appropriate box. 



Agree 
strongly 



The informaUon was 
presented clearly. 

The informaUon was 
provided at the 
appropriate level of detail. 

The instructor held my 
interest throughout. 



Neither 

agree nor Disagree 
/vgree Agree Disagree strongly^ 



The written materials 
were easy to understand. 



Any other comments? 
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The developers indicate that the form has proven useful to them in evaluating staff 
reactions to training, but they can provide no specific information about response rates or 
the reliability of individual items. They offer to share the results with the evaluation 
committee. At first, the committee believes the developers' fonn (reproduced above) is 
sufficient for its purposes. 

After further review, however, the committee decides that the developers' form does 
not capture the kinds of information it seeks. Specifically, the committee wants to find out 
how likely each trainee is to use the "Way to Go'' program. While the committee is 
interested in the quality of in-ser\ace instruction, it is more interested in the teachers' comfort 
with the new program and their initial willingness to implement its components. Even 
though the "Way to Go ' program has been mandated by the school board, committee 
members know that not all teachers will implement the program to the same degree. Further, 
it will be critical for the evaluators to know the identity of each respondent so that 
information on teachers' likelihood of implementation can be matched with attendance at 
training and, later, with their actual levels of implementation and their students' outcomes. 

In the end, the committee draws up its own set of short items (reproduced on the 
next page). It avoids putting teachers on the spot by asking them a yes-or-no question about 
whether they plan to implement "Way to Go." While short, this set of items will give the 
evaluators a preliminary idea of the context in which teachers are adopting the **Way to Go" 
program. Rather than a formal pretest, Ms. Brown tries out the questionnaire with a small 
group of teachers from another school who are receiving **Way to Go ' training in a different 
site. They make a few suggestions, and Ms. Brown is able to estimate that the questionnaire 
will take no longer than 10 minutes to complete. 

After approving the content, the committee is divided on when to administer the 
questionnaire. Several members feel that it would be best to administer the questionnaire at 
the end of the final day of training, to ensure a high response rate. Other com.mittee 
members point out that the participants will have to complete several **Way to Go * developer 
questionnaires during training and may resent the additional work. Further, if the teachers 
fill out the questionnaire in a group setting, they may feel inhibited about answering 
honestly. One member notes that it might be a good idea to let some time elapse after the 
training before asking teachers how they plan to use the curriculum. After all. at least 
several weeks will elapse before teachers actually begin to implement the program. 
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The committee drafts the following questionnaire: 
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Please indicate whether you agree or disagree 
"X" in the appropriate box. 



Wood County School District 
Teacher QuesUonnaire (Pretest) 

^vith each of the foUo^'ing statements by placing an 



Neither 

sTo^y Agree Agree Disagree_strongty 



Disagree 



1. 1 feel confident that I can 
implement the "Way to Go" 
program in my classes. 

2. 1 could use addiUonal help 
in implementing the -Way 
to Go" program. 

3. Although important, 
implementing "Way to Go" 
will take time from more 
important topics. 



4. Most of the students at my 
school have used alcohol 
during the past month. 

5. Veiy few of the students 
at my school have ever 
tried marijuana. 



1 3 4-6 7-9. 10-12. 13-15. 16-18, more than 18. 




Fonn Approved by Wood County School Distnct Evaluation Comm.ttee 



57 



After considerable discussion, the committee decides to administer the questionnaire 
at the beginning of the fall semester, 3 weeks after training ends. The evaluation committee 
decides that the benefits of in-person administration (higher response rate, uniformity in data 
collection point) are outweighed by the liabilities (teachers might give less valid answers). 
At the same time, later mail administration has its own advantages (the teachers will have 
had a chance to assimilate — or forget—training information, and confidentiality of response 
may be greater). The committee is aware, however, that obtaining a high response rate will 
require greater effort, 

A number of procedures are introduced to ensure the confidentiality of responses 
and to relieve teacher anxieties. First, Ms. Brown and the secretary, who will enter the 
teacher data in the computer, sign a statement pledging to maintain the confidentiality of 
teacher responses. The form they signed is displayed on the next page. Using the 
attendance sheets from the training, Ms. Brown identifies appropriate teachers with 10th- 
grade classes and assigns a random three-digit number to each teacher name. (The reader 
will recall that the evaluation committee decided earlier to focus the evaluation on 10th- 
grade students.) Ms. Brown creates a master list and then enters the three-digit codes on the 
blank questionnaires. 

There are other ways of ensuring confidentiality and decreasing respondent anxiety. 
If Wood County had engaged an outside evaluator, that person could have distributed the 
questionnaires either at the training or by mail, indicating that no one in the school system 
would know how any teacher had responded. Unfortunately, Ms. Brown cannot make that 
claim since she is a school district official. 

The survey is administered by internal mail during the first week of school. In their 
mail boxes at school, all appropriate teachers receive a personal cover letter from Ms. Brown, 
a precoded questionnaire, and a return envelope with Ms. Brown's name on it. The teachers 
are asked to fill out the questionnaire and return it within 1 week. 
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Ms. Brown and her seaetary sign this statement: 



OF co»ra>Bm«nT or 8»rvei data 

Sulrmcn^that arc to be strlcUy observed. 

Procedure, for lUinwrnuMs assurance of confldenUaUty. 

1 AU employees involved m surveys or evaluaUonsshaU sign this assurar^ 

1. Aueinpiuy'^ r^^„Hpnt<? all Information OF opinions 

survey data in their possession. 



survey data in tneir po»»^=»*""- 

JSol bemg «<.'B»e -^J " Z^p,m™ who « working ?» ^"P"*' 

assurance of conndenUaliiy. 



5. 



6. 



Signature 

Form Approved by wood county Schoo. District EvaluattonCommmee 




this cover letter accompanies the teacher questionnaire: 



Wood County School District 



. • • Woodville, US COOOO (555) 555-5555 

1650 Adminisirauon Blvd. wooavmc, ^ 



September 1, 1992 



Dear Mr. McCarthy, 

evaluating the ^on co-i"ee with representatives 

prevention program. ^^^^^^^^,^^3, and students is designing 
of teachers, parents, ' ^elp implement the 

and carrying out the evaluation ^o help^^ P^^^^ 

evaluation, we would ^^^^^ deceived in August and your plans 

in-service training you may have ^^ce ^^^^^ ^^^^ ^ 

,or using -e Program^ TheTn "ore^rhrrrc^estioLaire. Please 
^ftn-ourihe^^uestLraireven i. you did not attend the in- 
service training. 

r..p<.n>lblllcy CO -inc.in co„£id.„ti.Ucy . 

„o„i. you Pi.... *°^!::r°.ruSr'p°«"".. 
r..r;..-.r.r.s ry'^rr ... _ 

greatly appreciated. 



Thank you very much. 



Sincerely, 



El lie Brown 

Drug Program Coordinator 
Wood County Schools 



Enc . 
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About half of the teachers fail to return their questionnaires within 2 weeks 
(a common nonresponse rate for a mail questionnaire). To increase the response rate, the 
secretary calls teachers to remind them to send in the questionnaire (and sends them 
another form if they have lost the first). Teachers who still fail to respond are called agam 
and asked to respond to the information over the telephone (or they may be asked to 
make an appointment for a convenient time for a future telephone call to report the 
irrformation). Any teacher who refuses to respond is called by Ms. Brown in a last-ditch 
effort to obtain cooperation. 

Implementotion Question #3: Did Teachers Implement '^ay to Go"? 

To find out whether teachers do, in faa, implement the "Way to Go" program, they 
are asked about actual implementation toward the end of the school year. Once again, « ^ 
r^ecessary to administer the questionnaire to teachers with lOth-grade classes in health, 
physical education, and science to fulfill the requirements of the evaluation, regardless of 
whether they attended training. (Distria officials might decide to distribute the questionnaire 
more widely to find out about overall teacher implementation.) 

Items on the posttest are drafted to reflect those on the pretest, but they are more 
heavily focused on actual implementation. A sample of the posttest questionnaire appears 

on the following page. 

Obtaining a high teacher-response rate on the posttest is critical because « will be 
almost impossible to condua the student-level analysis without knowing whether the. 
teachers implemented the program. The committee decides to use the same administration 
procedure as for the pretest, but It instructs Ms. Brown to conduct an aggressive telephone 
follow-up to make sure that every- appropriate lOth-grade teacher is included. 



3a„ alicmativc ,o icachc-r Cima.cs of class ,«,ions devoted to "^ay to Go" might h..ve been an ongoing 
teacher recordkeeping system. 



The teacher posttest is administered at the end of the year. 



Please indicate whether you agree or 
"X" in the appropriate box. 



Wood County School District 
Teacher QuesUonnaire (Posttest) 

disagree with each of the foUowing statements by placing an 




^ee i agree nor I ! Disagree 

^ Agree ' Agr ee \ Disagree ; strongly 



strongly 



1. 1 felt confident in 

implementing the "^Way to 
Go" program in my classes. 



2. 1 could have used additional 
help in implementing the 
"Way to Go" program. 



3. Most of the students in my 
classes have used alcohol 
during the past month. 



4. Very few of the students 
In my classes have ever 
tried marijuana. 



implemented the "Way to Go" program, and estimate the 



Course name 



Number of class sessions 
devoted to **Way to Go" over 
school year 




I referred approximately (enter number). 
Any additional comments? 



students for drug or alcohol screening. 




Form 



Approved by Wood County School District Evaluation Comt >ittee 



Implementation Question #4: Was the New Disciplinary 
Policy Implemented? 



Finally, the Wood County evaluation committee wants to find out whether the new 
disciplinary policy is being enforced. To do so, it decides to compare drug-related 
suspensions and expulsions in the first year of "Way to Go" with those in the previous year. 
As soon as Ms. Brown begins data collection she discovers, however, that the district's 
summary statistics do not distinguish drug-related from other disciplinary actions. As a result, 
Ms. Brown and the secretary have to review each action and note whether drugs were 
involved. This task takes considerable time, and not all cases are clear cut (the records do 
not always reveal whether drugs were involved in the action). Based on an initial records 
review. Ms. Brown groups the data for both years using the following format. 



Wood County School District 
Student Disciplinary Record Form 
School Year 



Disposition of incident (final action) 



Nature of incident 
(describe, omitting names) 



Parent 
conf. 



Short 

Substance suspension 
abuse (one week 
assessment or less) 



Longer 
suspeasion 
(more than 

one week) Expulsion 



Other Unknown 



4 




Form Approved by Wood 



County School District Evaluation Committee 
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Ms. Brown and the secreiarv' could simply check off the number of drug-related 
incidents on a piece of paper, but that approach would not enable the committee to 
determine whether the disciplinary policy is being enforced. It also would not provide a 
record of the data collection process (i.e., the documentation) necessan^ to knov/ whether 
Ms. Brown has done a thorough and accurate job. To avoid this record-searching work next 
summer and to ensure accuracy in reporting, the evaluation committee recommends to the 
superintendent that student disciplinary records be amended to include a specific category 
for drug-related disciplinary^ actions. 

The Comparison School District 

Until now, this discussion has focused on evaluation activities in the Wood County 
School District. The reader may recall, however, that a subcommittee was appointed to 
approach a neighboring district about forming a comparison group for the evaluation. 
Fortunately, the neighboring district has agreed to cooperate. Because the comparison 
district is not implementing the "Way to Go" program, it will not be necessary to administer 
the *'Way to Go" implementation data collections in that district. Nonetheless, the Wood 
County evaluators wiW want to know something about the nature and extent of drug 
education for lOth-grade students in the comparison site. If teachers and district officials in 
that site are willing, the comparison teachers likely to teach drug education (these will 
depend on the district and school) could answer a brief questionnaire similar to the posttest 
for the treatment teachers. 
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This questionnaire was designed for teachers at the comparison site: 



End of School Year 
Comparison Site Teacher (Posttest) QuesUomiaire 



Please Ust the courses in which you taught drug 
instruction you provided this school year. 



education and the nature and amount of 



Drug education curriculum 
or approach used 



Number of class sessions 
devoted to drug education 
over school year 




the appropriate box. 



Neither 

agree nor ; \ Dl^^ 

Agree Agree • Disagree • strongly 



Agree ' agree nor 

strongly 



1. Most of the students in my 
classes have used alcohol 
during the past month. 

2 Very few of the students 
in my classes have ever 
tried marijuana. 



Have you ever used the "Way 



toGo^drugeducationprogram? {Circle one) Yes No 

-Way to Go" program? {circle one) Yes No 



Do you currently use any components of the " 
during this school year? 

,C^.o«l Ves NO ,fyes.p.as.in.>ca«nun,b.,ofstu.e„urer».- 
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Asking questions such as these will enable the Wood County evaluation conimittee 
to know whether students in the comparison site get roughly equivalent amounts and types 
of drug education. It will also enable the comparison district to know more about the 
amount and nature of the drag education it provides. Finally, it is important to know 
whether any of the comparison teachers are using "Way to Go" components in order to 
conduct a more accurate analysis of student outcomes. 

In the comparison district, the teacher questionnaire is administered at the same time 
as the treatment distria teacher posttest. The procedure is similar to that outlined for the 
treatment district, although the cover letter is somewhat different and is signed by an official 
of the comparison district. 

Evaluating Student Outcomes 

The committee designed its own instruments for evaluating the implementation of 
the "Way to Go" program because the program's combination of activities sufficiently differed 
from other programs that no existing source could provide a complete model. The 
commirtc. oorrowed ideas and questions for measuring the implementation of different 
program components from a variety of sources but merged them to make them fit the 
evaluation design. The instrument to measure program outcomes will be developed 
differently because the goals of the program— a reduaion in student drug use, a delay in first 
use by students, and an increase in negative attitudes toward drugs— have been widely 
studied. As a result, many evaluators have developed measures to address these objectives, 
and there is a wide range of existing methods and measures from which to choose. 

There are several advantages to using pre-existing methods and survey instruments 
(questionnaires) including the established validity and reliability of items, and the availability 
of existing comparison groups (based upon results derived from other administrations of the 
same instrument or individual items). There are also some issues to consider in using pre- 
existing survey instruments, including: 

♦ Cost, as copyrighted surveys may have to be purchased; 

4 Ensuring validity, as the items may not specifically address the 
evaluation's concerns; 

# Ensuring appropriate comparisons, as the charaaeristics of other groups that 
have used the same instrument or items may be quite different from those of 
participants in the program being evaluated. 
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The Wood County evaluation committee decides to review some of the most widely 
known questionnaires about student drug use. While reviewing these instruments, the 
committee keeps in mind that the Wood County evaluation will need to measure students- 
knowledge of drugs, attitudes toward drugs, and actual drug use, and that the survey will be 
limited to lOth-grade students. As mentioned earlier, it will be necessary' to administer a 
questionnaire to Wood County 10th graders both before and after the implementation of the 
"Way to Go" program and to administer the same pretest and posttest to a comparison group 
of lOt-graders in the neighboring district. The questionnaire will have to gather student 
background information-such as race, gender, age, and educational attainment of parents- 
that will enable the committee to determine whether the two groups of students are 
comparable. The committee also hopes to use the student questionnaire to supplement 
the teacher data by providing the students' point of view on the implementation of the 
district's disciplinarv policy and drug education curriculum. The section that follows 
provides examples of possible survey items for each Wood County objective drawn from 
well-known questionnaires. 

The examples in this section are not intended to serve as a model questionnaire. 
There are two reasons the handbook is not providing a model, or even a set of instruments 
fiom which readers may choose. First, any questionnaire developed for the handbook 
would have borrowed heavily from existing questionnaires, so it would not have undergone 
the necessarv validity and reliability checks. Second, no questionnaire provided here 
would be applicable to all districts or schools, so even if an instrument had been developed 
(and tested) it would still need to be altered by local evaluators. Instead, the following 
section presents sample items, and the appendix lists compendia of questionnaires. 
These compendia include information on the validity and reliability of a wide range of 
individual instruments. 

Rates of Drug Use 

Ms Brown and the committee have little trouble identifying appropriate questions 
on drug use in existing surveys. Even so. they must choose which drugs they want to ask 
about and specify the frequency and type of use. At the same time, they want to keep 
respondent burden to a minimum. They consider the following items. 
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Alternative One: from Monitoring the Future 



1. on how many occasions have you had alcohoUc beverages to drink... 
(Mark one circle for each line.) 



a. in your lifetime? 

b. during the last 12 months? 

c. during the last 30 days? 
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0 1-2 i 3-5 i 6-9 
Occasions i Occasions I Occasions i Occasions 


10-19 
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20-39 
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o 
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o 
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o 


1 
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\ o 
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o o 


o 


o 


o 


o 


o 



.V, TA<^9WFFKS How many times have you had five or more drinks in a row? 

ONone O 3 to 5 times 

OOnce 0 6to9Umes 

O-mce O 10 or more times 

3. on how many ocaisions (if any) have you been drunk or very high from drinking 

alcoholic beverages? , i ' 

Q 5 ' 6-9 ' 10-19 ' 20-39 
occasions . Oc^lns , Oa^'slons , Occasions , Occasions , Occasions , 40. 



au in your UfeOmc? 

b. during the last 12 months? 

c. during the last 30 days? 



O o 



o 



o o o 



O 0 



o , o o 
o o ■ o 
o o o 



^Monitoring the Future includes similar items for cigarettes, marijuana. LSD. cocaine, "crack" cocaine, 
amphetamines, barbiturates, narcotics, inhalants, and steroid.s. 
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Alternative Two: from the Youth Risk Behavior Survey 



During the past 30 days, on how many days did you 
at least one drink of alcohol? 

O Odays 
O lor 2 days 
O 3 to 5 days 
O 6 to 9 days 
O 10 to 19 days 
O 20 to 29 days 
O All 30 days 



O Odays 
O lor 2 days 
O 3 to 5 days 
O 6 to 9 days 
O 10 to 19 days 
O 20 or more days 



5The Youth Risk Behavior Sun^ey also includes similar items for cigarettes, marijuana, steroids, cocaine. 



2 During the past 30 days, on ho) 
more drinks of alcohol in a row 




and other drugs. 
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First Use of Drugs 



Because the "Way to Go" curriculum seeks not only to reduce drug use but also to 
delay first-time use, the committee could include the following questions. 



From Monitoring the Future: 



V, «f thP fnllowinfi things? Don't count anything you 
When (if everl did you FiRST do each of the foUowmg mmg 
took because a doctor told you to. (Mark one circle for each line.) 



a. Smoke your first cigarette 

b. Smoke cigarettes on a daily basis 

c. Try an alcohoUc beverage— more 
than just a few sips 

d. Drink enough to feel drunk or 
very high 

e. Try marijuana or hashish 

f. Try "crack" cocaine 

g. Try cocaine in powder form 

h. Sniff glue, gases, or sprays to 
get high 

i. Try steroids (anabolic steroids] 



GRADE LEVEL 


i Never 


4th or 
below 


5th 


6th ; 


7th 


8th . 


9th . 


10th 


o 


o 


■ O 


o . 


o 


O ' 


^ ! 


o 


0 


o 


o 


o 


0 


0 


o 


o 


o 


0 


o 


o 


0 


o 


! O 


o 



o 


0 


o 


o 


0 


o 


o 


o 


o 


0 


o 


0 


o 


0 1 




o 


o 


o 


o 


0 


0 


o 


o 


o 


o 


0 


0 


o 


o 


o 


o 


o 


0 


0 


D 


o 


D 


o 


0 


^ o 


o 


0 


o 


o 


o 


0 


0 


c 
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intent to Use Drugs 



In addition to asking students questions about their actual drug use. the committee 
could ask students if they intend to use drugs because this t%'pe of question helps to predict 
future drug use. The committee identifies a number of useRiI items in the Youth Risk 
Behavior Survey including the following question. 



From the Youth Risk Behavior Survey: 



Ooyouthmkyou..Ut...garettes.okin,du.ngthenextI2months. 

3 I have already tried cigarette smoking 
0 Yes. 1 think 1 ^vill try cigarette smoking during the next 
12 months 

3 NO. I think 1 Will not try cigarette smoking during the 
next 12 months 



A different dimension of intent is measured in this item from the Prince George's County, 
MD. Dnig Interest Survey, ^\-hich asks students how they would respond if offered daigs. 

From the Prince George's County, MD, Drug Interest Survey: 



If I were 



offered been wine, or liquor. I would: 



D Say yes 

3 Maybe say yes 

3 I don t know what rd do 

O Maybe say no 
■;) Say no 



i 
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Attitudes Toward Drugs 



Because the "Way to Go" curriculum seeks to increase students' negative attitudes 
toward daig use. the committee might consider using the following item. 



From Monitoring the Future: 



disapprove of people doing each 



of the following (Mark one circle for each line.) 



DiSp^ove Disapprove DlsappVo've Unfarnlliar 



Can't Say. 
Strongly Drug 



a. Smoking one or more packs of 


D 


0 


O 


O 


cigarettes per day 










b. Using smokeless tobacco regularly 


D 


D 


D 


O 


c. Trying marijuana once or twice 


D 




3 


3 


d. Smoking marijuana occasionaUy 




0 


D 


0 


e. Smoking marijuana regularly 




0 




0 


f. Trying cocaine in powder form once 




3 


D 


J 


or twice 










g. Taking cocaine powder occasionally 




0 




D 


h. Trying ''crack" cocaine once or twice 






!) 


0 


i. Taking -crack" cocaine occasionally 


D 




) 


3 
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Perceived Risks 



Because the "Way to Go" curriculum seeks to increase student knowledge about the 
risks involved in using drugs, Ms. Brown wants to include an item that measures student 
perceptions of those risks. 



From Monitoring the Future: 



(physically or in other ways) if they... 













Can't Say. 




No 
Risk 


Slight 
Risk 


Moderate 
Risk 


Great 
Risk 


Drug 
Unfamiliar 


a. Smoke one or more packs of 


o 

0 


D 


O 


o 


o 


cigarettes per day 
b. Use smokeless tobacco regularly 


O 


o 


o 


o 


c. Try marijuana once or twice 

d. Smoke marijuana occasionaUy 


o 
o 


O 
0 


o 

D 


o 

0 


o 

0 


e. Smoke marijuana regularly 


o 


D 


0 


o 


o 


f. Tr>^ cocaine in powder form once 


0 


0 


0 


3 


3 


or twice 










g. Take cocaine powder occasionally 


0 


O 


0 


O 


3 


h. Try -crack- cocaine once or twice 


0 


0 


O 


O 


o 


1. Take "crack" cocaine occasionally 


) 


) 


D 


0 


3 



i 

A 
1 
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The committee might also use this alternative question on perceived risks. 



From the Prince George 's County, MD, Drug Interest Survey: 

■3 



Using drugs is not really dangerous. ^ 

'1 

O Very true i 
O Sort of true '-^ 
O Neither true nor false 

O Sort of false 
O Very false 



This suivey includes similar questions on the specific risks of drinking, glue sniffing, using 
steroids, and smoking marijuana and PGR 

Drug Education 

The teacher survey measures implementation of the 'Way to Go" curriculum from 
the teachers' perspective, but it is useful to measure implementation from the students' 
perspective as well. The committee believes that student viewpoints will enhance their 
understanding of the implementation of the curricular component of the "Way to Go" 
program. Generally speaking, multiple measures provide more confidence in evaluation 
findings than does a single measure. Furthermore, Ms. Brown argues that the committee 
should not put all of its data "eggs in one basket." Perhaps several teachers complain about 
the curriculum during the school year and then refcse to fill out the teacher posttest. 
The inclusion of a student measure of program implementation will help to ensure that 
the committee will have some indicator of program implementation to compare with 
student outcomes. 
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From Monitoring the Future: 



1. Hav.youhada„,drug«.u»tto„co>,«sor.ecture,«s*»,, (Mar.o„,J 

O No 

S ;:Ct^«d P.ea« a^er *e .ono^ng ,u,stto„s., 

a. WouW you say that *e mformauon about d™gs that you «celv,d i„ school 
classes or programs has... 

O Made you less interested in trying drugs. 

O Not changed vour interest in trying drugs. 
0 Made you more interested in trying drugs. 

3. Howmanyofthefollo^vingdrugeducauonexperienceshaveyouhadinschool? 

(Mark all that apply.) 

n A snecial course about drugs 

O niTs lectures, or discussions in one of my regular courses 
O FUms or lectures outside of my regular courses 
O Special discussions or group sessions about drugs 

4. overall, how valuable were the drug educaUon experiences to you? 

D Little or no value 
O Some value 
3 Considerable value 
3 Great value 



1. 



These que.stions would have to be altered slightly in Wood County to make it clear that this 
years dnig education is the focus of the inquir\'. 

Discipiinory Environment 

The new disciplinarv- policy is an important component of the "Way to Go" program, 
so the committee w ants to find out if students know aliout the policy and if the policy is 
reducing the availability of dmgs in school. 
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The following item is from Monitoring the Future: 



if you wanted to? (Mark one circle for each line,) 



a. Cigarettes 

b. Alcohol 

c. Marijuana (pot grass) 

d. LSD 

e. PCP (angel dust) 

f. Amphetamines (uppers, 
pep pUls. bennies, speed) 

g. Barbiturates (downers, 
reds, yellows) 



Can't Say. 

Drug Probably ; 
Unfamiliar. Impossible : 


Very 
Difficult 


Fairly 
Difficult 


Fairly 
Easy 


Very 
Easy 


0 


o 


o 


o 


O 


0 


0 


o 


0 


o 


0 


o 


3 


o 


o 


o 


o 


o 


0 


o 


o 


o 


0 


o 




o 


o 


o 


o 


o 



o 



o 



At first glance, the committee thinks the above question is appropriate, but then a 
member points out that it is intended to measure general drug availability rather than at- 
school drug availability. The committee member offers an alternative item from the 
National Crime Survey—School Crime Supplement, that may he better suited to 
examining in-school use. 
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From the National Crime Survey— School Crime Supplement: 



HOW easy or hard is it for someone to get the foUowlng things at your school? 
(Mark one circle on each line.) 



Easv 



Don't Don't Know 
Hard Impossible_^_Know J^^^g 



Alcoholic beverages 


0 


O 


o 


O 


0 








0 


0 


0 


Marijuana 














0 




0 


D 


D 


Cocaine 














o 


0 


0 


O 


Uppers/downers 










Other illegal drugs 


0 


o 




O 


0 











This item could help to gauge changes in student perceptions of drug availability at school, 
because a reduction in the availability of dnigs at school is clearly one desired outcome of 
the new disciplinary policy and a possible intermediate step toward the reduction of student 
daig use. 

Ms. Brown also identifies a number of items in the National Crime Survey- 
School Crime Suppletnent that could help to measure student awareness of the new 
disciplinary policy, including the following. 
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From the National Crime Survey— School Crime Supplement: 



V^har happe™ .. a s.„d»t who ge,B caugK, domg *e MlowWg »■ yo™ school, 
(Mark all choices that apply.) 



Being disrespectful to teachers? 



Drinking or being drunk at school? 



O Nothing 

O Student disciplined by teacher 

O Student sent to the principal's office 

O Student's parents are noUfied 

O DetenUon 

O Suspension 

O Other— Specify 

3 Don t Know 

O Nothing 

O Student disciplined by teacher 

O Student sent to the principal's office 

O Student's parents are noUfied 

O Detention 
O Suspension 

O Other— Specify . 

O Don't Know 



Of course, this item includes a behavior that is not directly related to drug use. Furthermore, 
the item does not ask about the disciplinar\' actions taken as a result of drug use or 
possession. .Ms. Brown decides to modify the item, yet retain its format, by adding the 
following categon- and choices. 
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Ms. Brown's substitute question: 



. . hnc^etscaufiht doing the following things m your 
What happens to a student who gets caught ao g 

school? (Mark all choices that apply.) 



Possessing illegal drugs at school? 



0 
O 

o 
o 

0 
3 



Nothing 

Student disciplined by teacher 

Student sent to the principal's office 

Student's parents are noUfted 

Detention 

Suspension 

Expulsion 

Other— Specify 

Don't Know 



Responses to this question can help to determine what proportion of students know 
about the school's polic>'. Findings about student knowledge of the disciplinary policy 
could be reported as an intermediate outcome in the evaluation because student knowledge 
is not only a means to an end (preventing drug use at school), but may be a critical 
intermediate step. 
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To gauge the general disciplinarv' climate at home as well as at school, the 
committee could also consider the following item. 



From the Prince George's County, MD, Drug Interest Survey: 



If your parents knew you were smoking cigarettes, do you 
think vou would get in trouble at home? 



:3 Yes 
0 Probably 
O I'm not sure 
O Probably not 
O No 



If your parents knew you were drinking beer, ^^ne. or liquor, 
do Jou think you would get in trouble at home? 



O Yes 

J Probably 

O I'm not sure 

3 Probably not 

0 No 



SO 

V 



Drag Availability and Acceptability 



One of the goals of the "Way to Go" program is to create a normative climate that 
makes drug use unacceptable. To measure whether the curriculum is having the desired 
effects on the school environment, the committee could include the following item. 



From ManUartng the Future: 



How many of your friends would you estimate... 



a. Smoke cigarettes? 

b. Use smokeless tobacco? 

c. Drink alcoholic beverages 
(liquor, been wine)? 

d. Get drunk at least once a week? 

e. Smoke marijuana or hashish? 



None 


A Few 


Some 


Most 


All 


O 


O 


O 


O 


O 


O 


0 


O 


O 


0 

] 


o 


o 


0 


. o 


o 




3 


0 


0 


o 


0 


0 


3 


o 


0 



ERIC 



81 



The conuniiiee might also use the following items: 



From the Prince George's County, MD, Drug Interest Survey: 



L HOW often are you around kids who smoke cigarettes? 



O Never 

O Almost never 

O Sometimes 

O A lot 

O Always 



2. How often are you around kids who drinK beer. wine, or Uquox^ 



O Never 

O Almost never 

O Sometimes 

O A lot 

O Always 
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StHdefit Background 

Demographic and other background information will help the committee to 
determine whether subgroups of students respond differently to "Way to Go.'* The 
demograpliic data will also be used to determine if the comparison group of 10th graders in 
the neighboring school district does, in fact, resemble Wood County's 10th graders. 
Demographic information could also be used to "target" . omponents of the "Way to Go" 
program more closely to groups of students who appear more likely to use daigs. 
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Background questions from Monitoring tbe Future: 



How old were you on your last birthday? 



on years old or less 
O 12 years old 
O 13 years old 
O 14 years old 

What is your sex? 
O Male 



O 15 years old 
O 16 years old 
O 17 years old 
O 18 years old or more 



O Female 



. How do you describe yourself? 

O Native American or American Indian 

O Black or African American 

O Mexican American or Chicano 

O Cuban American 

O Puerto Rican American 

O Other Latin American 

O Oriental or Asian American 

O White or Caucasian 

O Other 

What was U,e Orst language you fe^ed to ^ «h.„ you were a child, 
. (Mark one.) 

O English 
O Spanish 

O Some other language 

5. Where are you living now? 

O On a farm 

O In the country, not on a farm 
O In a city or town 

6. Which of the following people live in the san.e household with you? 

(Mark all that apply.) 



O Fatlier (or stepfather) 
O Mother (or stepmother) 
O Brothers (or stepbrothers) 
O Sisters (or stepsisters) 



O Grandparent(s) 
O Other relaUve{s) 
O Non-relatlve(s) 
O I live alone 
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7 Onaverage,howmuchUmedoyouspendafterschooleachdayath^ 
with no adult present? (Mark one,) 



O None or almost none 
O Less than 1 hour 
O 1-2 hours 



O 2-3 hours 

O More than 3 hours 



8. What is the 



highest level of schooling your father completed? 



O Completed grade school or less 

O Some high school 

O Completed high school 

O Some college 

O Completed coUege 

O Graduate or professional school after coUege 
O Don't know, or does not apply 



9. What is the 



highest level of schooling your mother completed? 



O Completed grade school or less 

3 Some high school 

O Completed high school 

3 Some college 

O Completed coUege 

O Graduate or professional school after college 
0 Don t know, or does not apply 

10. Does your mother have a paid job? 

D No 

3 Yes. part-Umejob 
3 Yes, fuU-timejob 
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Finally, students are asked to name their science, health, and physical education teachers for 
the current semester. 



Administration of tlie Student Questionnaires 

Although the committee has now identified useful items from a range of surveys, it 
wants to keep respondent burden — the amount of time it takes to fill out the questionnaire — 
to a minimum. This is true for both Wood County students and perhaps even truer for those 
in the comparison site. In addition. Ms. Brown knows that she will have to compile and 
analyze all of the data, so she wants to keep the questionnaires as short as possible. There is 
no perfect solution here, just a reminder to set questionnaire time limits, clarify subject 
matter priorities, and stick with tbem. 

After heated committee debate about which items to include, the student question- 
naire is kept to a length that members estimate can be completed in half an hour. An 
English teacher reviews the items and estimates that they can be read with good comprehen- 
sion by almost all lOth-grade students. At this point. Ms. Brown conducts a pilot test of the 
questionnaire with a summer school class composed of students who will be entering the 
Uth grade in the fall. There are no identifiers on the questionnaire so that the students' 
responses will remain anonymous. The students are encouraged to note places where 
changes in language might make the items more appropriate for high school students. 
Ms. Brown reports to the committee that the students took an average of 40 minutes to com- 
plete the questionnaire. Several students recommended that the actual questionnaire include 
a cover page that reassures students about the confidentiality of their responses. Ms. Brown 
makes several revisions to shorten the student instrument. She also analyzes the individual 
responses on the pilot test for reliability and then destroys the pilot test questionnaires. 

Preparations are begun for administration of the fall pretest. Because the question- 
naire is only for lOth-grade students, the committee decides that the best place to administer 
the survey is in lOth-grade homerooms. (The idea of a mail questionnaire for 10th graders is 
rejected, as the committee believes that the nonresponse rate would be too great.) It is fur- 
ther decided that, to assure students about confidentialit^^ the questionnaire will be passed 
out and collected by someone other than the regular homeroom teacher. Teachers who 
rarely come in contact with lOth-grade students (e.g., teachers of advanced science or math 
courses) will be asked to switch homerooms with the lOth-grade homeroom teachers. In this 
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way, the persons administering the 

questionnaire will be unfamiliar to 

the students. (Homeroom lasts only 

45 minutes so it is critical that the 

questionnaire be short enough to be 

completed in that period.) The 

committee corisiders and rejects the 

idea of administering the question- 
naire to an assembly of lOth-grade 

students. Several members believe 

that in the assembly setting many 

students would discuss answers with 

each other, and it m-jht be difficult 

to ensure the return of all surveys. 
Ms. Brown abides by 

appropriate Federal, state, and 

district rules for parental consent to 

administer questionnaires to students. During the first week of school, she mails a letter to 

the parents of all lOth-grade students over the superintendent's signature, explaining the 
new program and the evaluation. She indicates that students will be surveyed before and 
after program implementation, and that the survey will ask questions about drug use. She 
notes that responses will be held in strict confidence. Parents are asked to indicate whether 
they will permit their children to participate in the evaluation by signing an attached form 
and mailing it to school in a self-addressed, stamped envelope. The same procedure is 
followed in the comparison district. 

After follow-up phone calls to nonrespondents, a small number of parents asks that 
their children be excused from the survey. If resources were available, the committee could 
analyze the charaaeristics of these 10th graders to see how they compare with participants 
along such dimensions as race, gender, and grade point average— data that are already 
available through the school system. During analysis, Ms. Brown and the committee will 
want to compare actual respondents with aggregate information on the lOth-grade student 
body to see whether the respondents reflect the overall group. 

To assure confidentiality (and a discreet questionnaire administration) in 
homerooms, students will not be asked to sign their names or provide other identifying 



ERIC 



information on their questionnaires. It is decided that teachers who will administer the 
questionnaire will be given a set of forms with precoded student identification numbers 
(random numbers matched to actual numbers on a master list retained by Ms. Brown). Each 
questionnaire will have a removable self-stick note with the student name attached. As the 
teacher calls the narae, he or she will hand the student the survey and remove and discard 
the self-stick note. Students will also be provided with blank envelopes in which to place 
their completed questionnaires before they drop them in a box at the back of the room. 
Fifteen minutes before the end of the administration period, students will be asked to fill out 
the individual background items, if they have not already done so. Students whose parents 
have asked that they be excluded will be treated in a way that does not call attention to 
them. They will be given a different questionnaire that asks nothing about drugs and has no 
identifying marks. 

The committee is concerned that students who are absent the day of questionnaire 
administration be given an opportunity to complete a survey. It is decided that students who 
are not present at homeroom the day of questionnaire administration (or who fail to return 
their questionnaires) will be given the questionnaire in another class during the following 
week and given time to fill it out. The teacher in that class will be asked to distribute and 
collect the questionnaires. Nonetheless, the committee is aware that some students, possibly 
those most likely to use drugs, may still not fill out questionnaires. Committee members also 
know that some responses will probably be unusable because the students will answer items 
incorrealy or leave too many answers blank. The committee concludes, however, that there 
is no inexpensive way to ensure 100 percent participation. 

Aside from increasing the likelihood of a high response rate, another reason to 
avoid a long questionnaire is to make it relatively easy for Ms. Brown and her assistant to 
enter responses into the computer. One way to enter a large number of responses is to 
develop a machine-readable answer format. Once questionnaires are received. Ms. Brown 
and her assistant use a simple computer program to enter student answers. They also merge 
each student s questionnaire response with data on his or her course schedule (already 
computerized, in most districts) that will enable them to match the student to his or her 
health, physical education, and science teachers. 

Administration of the posttest is conducted in the same manner. The same 
questionnaire is used. While use of the same items is sometimes unwise—answers on the 
pretest can influence answers on the posttest--4he sizable period of time that has elapsed 
(fall to late spring) makes it unlikely that students will recall their previous responses. 
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Findings From Woovd County 



This section provides sample findings derived from the Wood County data 
collections. The presentation of findings demonstrates how ixiformation can be aggregated 
and displayed. The presentation uses both narrative and graphic displays. None of the 
findings reported in rhis section requires sophisticated statistical analysis. The first findings 
are presented narratively, the later findings as tables and graphs. 

Attendonce nt in-Service Troining 

Based on the attendance sheets collected from the training sessions, Ms. Brown 
produced the following data summary: 

♦ Of 25 health, physical education, and science teachers scheduled to attend 
the week-long staff training for the "Way to Go" curriculum, 18 attended all 
five sessions, 4 missed one session, and 3 failed to attend. 

❖ Ail 3 health teachers attended the full week, as did 9 of 10 physical 
education teacheis. Of the science teachers, 6 attended die full week, 3 
missed one session, and 3 did not attend. 



In short, most teachers scheduled to receive the training did attend. A disproportionate share 
of those who did not attend were science teachers. 



Staff Uicelihood of impiementing "^ay to Go'' 

The pretest was administered to all 25 teachers scheduled to implement the "Way to 
Go" curriculum with lOth-grade classes. After mail and telephone follow-up, 24 teachers 
returned pretest questionnaires. The results showed that: 

♦ Most teachers (17) agreed with the statement "I feel confident that I can 
implement the 'Way to Go' program in my classes." Of those teachers, 10 
agreed strongly. Only two teachers disagreed with the statement (none 
strongly). The rest were neutral. There were no major differences by 
subjea taught (health, science, or physical education). 

♦ Six of the 24 teachers agreed or agreed strongly with the statement 
"Although important, implementing 'Way to Go* will take tkne from more 
important topics." Five of the sbc were science teachers (including the three 
who did not attend training). 
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Based on these findings, Ms. Brown concludes that a high overall level of implementation 
appears likely but that a subset of science teachers might not implement the program fully. 

Teacher Implementation of ''Way to Go'' 

The results of the teacher posttest appeared to bear out Ms. Brown's initial concerns. 
Here is a display that shows the number of 50-minute sessions devoted to "Way to Go" by 
the 23 teachers who returned the posttest. (Incidentally, staff changes during the year 
resulted in posttests returned by two teachers who had not participated in training or in the 
pretest. Ms. Brown elected to keep the two teachers in the end-of-year findings because she 
wanted to link their responses with those of their students.) 



Teacher reports of number of 50-nunute class sessions 
per school year devoted to "Way to Go 
(n = 23) 



1 



Average number 




of class sessions 
devoted to 

Teachers subject are_a_ 


Range in number 
of class sessions 




4.5 

Science 


0-12 


6 2 

Physical education 


3-10 


9 9 

Health ' 


8-12 


Source: Wood County Teacher Suivey 
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Enforcement of the New Disciplinary Policy 



As Ms. Brown had suspected at the outset of the evaluation, interpreting the 
information on disciplinary actions proved difficuii. Here are the results of the review 
of records. 



Drug-related disciplinary actions at Wood County High School, 
by type of action and school year 



Source: Wood County Student Disciplinary Records 



Final action 


1991-92 


1992-93 


Parent conference \ 


7 


9 


Drug assessment | 


0 i 


8 


i 

Short suspension i 


2 






s 


(1 week or less) i 






Long suspension ! 


0 


i 3 






(more than 1 week) j 








2 


2 


Expulsion 






Referrals: PoUce/arrests 


2 


2 



'"i 



1 
i 



There was some increase in disciplinary actions in the first year of the program. The 
clearest change was the increase in the number of substance abuse assessments, a procedure 
that was adopted as part of the new policy. Ms. Brown argued that the increase in the use 
of the assessment procedure was itself an indication that the policy had been implemented. 
She concluded that it would take another year or more to see the effect of the policy on 
dnjg-related behavior at school. 
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Student Outcomes 



Most of the evaluation findings focused on student behavior. This section presents 
several tables to demonstrate how outcome findings may be displayed. 

Comparison Between Wood County and Comparison District 

This table displays a behavioral comparison between Wood County and the 
neighboring district. 



Percentage 



of 10th grade students who reported beUig 



4«*fc^ umt 30 dm. piiw to and tfter 
^-««%v at iftMt once to tnc i»i « , , 

XSieS.ti«>» of the "Way to Go" progran.. by school 



I 


— 

Wood County 
i Higji School 
(n=:249) 


! Comparison 

school 
1 (n=182) 

J , 


FaU 1992 

(Before "^Way to Gol 


23% 


i 

i 28% 
i 


Spring 1993 
(After "^Way to Go") 


27% 


38% 


Percentage difference 


+17% 


' +36% 





Source: 



: wood county and Comparison District Student Surveys 



trf^-.-^!-•t^--.2_■ 



This table shows that lOth-grade students at Wood County High School were less 
likely to report being dmnk prior to the implementation of the program than were students 
at the comparison school. At the end of the year, fewer additional Wood County students 
reported being dmnk. The results are encouraging for the "Way to Go" program. It should 
be noted, however, that the initial differences in the two groups make definitive statements 
about the results of "Way to Go" difficult. 
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With respea to lifetime marijuana use, the positive results for the "Way to Go" 
program are clearer. (The results are displayed below in a table and as a bar chart.) The 
initial use levels at the two schools were closer, so the differences in changes are heartening. 
It should be noted, however, that the numbers of students who have ever tried marijuana are 
quite small, so any changes appear large and slight differences can be exaggerated. 



Percentage of 10th grade students who ^f^^^J^^' 
^ 2il^ prior to «d after iinplem«Utlon 

of the "^ay to Go" program, by school 



Source: 



: wood County and Comparison District Student Sun/eys 



5- 


Wood County 
Higji School . 


Comparison 
school 


Fall 1992 

(Before "Way to Go") 


9% j 

1 


11% 


Spring 1993 
(After "Way to Gol 

Percentage difference 


12% 

1 +33% 


17% 
+55% 

i 



1 



Percentage of 1 0th grade students who reported ever tiying marijuana, prior to ond 
ofter imptementation of the 'V.'Qy to Go" program, by school 

20Xt 

17X 



1SX 



of lOtthgnxie 
students who 10X 
everHiod 

mofijuana 

5X 



0% 




foll992 



CMftrbM Distrkt 
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Even fewer students have ever tried cocaine, so ^hat interpreting small changes 
cocaine use is more difficult. 



Percentage of 10th grade students who reported ever 
cocaine prior to and after implemenUtlon 

of the -Way to Go" program, by school 



Fan 1992 

(Before "Way to Gol 

Spring 1993 
(After "Way to Gol 

Percentage difference 



Wood County 
High School 



4% 

4% 
No change 



Comparison 
school 



2% 



3% 
+50% 



source: Wood County and Comparison District Student Surveys 



Percentoge of 1 0th grade students who reported ever trying cocaine prior to and 
ofter implementation of the "Way to Go" program, by school 




HCMforhoa District 



Voriotion Within Wood County High School 



The next displays show ''Way to Go" behavior effects for Wood County students 
exposed to varying amounts of instruction. It should be noted that health and physical 
education are eiectives at the lOth-grade level in Wood County, so student exposure will 
vary, and those students who elect to take health or physical education may differ in some 
(unknown) ways from those who do not. To ascertain exposure levels, Ms. Brown grouped 
•the data in the following ways. 



Amount of 
exposure (number of 
courses with -Way to: 
Go" components) 


Number 
of students 


Fall {Before 
-Way to Go"), 
percent 
of students 


Spring (After 
-Way to Go"), 
percent 
of students 


Percentage 
' difference 


0 courses 


29 


31% 


36% 


'■ +16% 

1 


1 


69 


20% 


: 23% 


+15% 

i 


2 


104 


22% 


24% 


' +9% 


3 or more 


65 


25% 


23% 


-8% 



source: wood County and Comparison District Student Sun,eys^ 



Several observations are in order. First, it appears that some students did not 
receive any exposure to "Way to Go." and that those students had a higher initial likelihood 
of getting dmnk in the past 30 days. The rate of increase in getting dmnk from fall to spring 
was essentially the same for students with no exposure at all and those with exposure in 
only one course. Students with exposure in two courses reported being dmnk in the past 30 
days more often after the program, but their rate of increase was smaller than that of students 
with less exposure. Only students with exposure in three or more courses showed actual 
declines over the period. 
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Ms. Brown also reported on behavior effects in relation to hours of "Way to Go" 
instruction (total exposure). Below is the table she used to display those data. 



reported having been drunk at least once 
after! 



Hours of 
-Way to Go" 
InstrucUon 


Number 
of students 


Fall 
(Before 
-Way to Go") 


No instruction 


29 


31% 


MO 


40 


25% 


11-20 


109 


24% 


21-30 


71 


25% 



Spring 




(After 


Percentage 


-Way to Go") 


difference 


36% 


+16% 


29% 


+ 16% 


26% 


+8% 


25% 


0 



Source: Wood County Student Suiveys 



Based on these data, Ms. Brown reports to the board that total exposure time appears to play 
an important role in the effectiveness of the "Way to Go" program. Of course, it may also be 
that students who elect to take health and physical education have other characteristics that 
help to determine the observed effects. There is no way to know for ceitam. The data also 
show that some of the students with the highest reported alcohol intake are not being 
reached by the program, probably because they do not elect health or physical education 
and their science instaictors are the ones least likely to implement "Way to Go." 
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Attitudes Toword Drug Use 



Ms. Brown also chose to report on changes in attitudes and perceptions of risk as a 
result of the program. Here are two sets of displays that compare Wood County and its 
neighboring distria. The first set shows data on negative attitudes toward alcohol use before 
and after the program. 



Percentage of lOth-grade students who dlstpptorcd or 





Wood County 
Hlgjti School 
(n = 249) 


Comparison 
school 
(n=182) 


FaU1992 

(Before "Way to Go") 


35% 


40% 


Spring 1993 
(After "Way to Go") 


47% 


44% 


Percentage difference 


+34% 

. — ' 


+10% 



Source: Wood County and Comparison District Student Surveys 



Pefcentoge of 1 Otfvgrade students who disopproved or strongly disopproved of twving five or more drinks 
inarow,priortoandaftefimplefncntotionofti)e'Woyto6^^ propm, by school 



50% 

PafCiotoge 40X 

oflOth^ 
studentswho 30% 
disopprovMl of 
homgfive 20% 
ormoredfinb 
inarow 

10% 
0% 




Fdil992 
Mm 1^0^ to 60' 



SpfingmS 
Jifm'WofioGo'' 



iWMdCeMty 



Dbtricf 
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These data show changes in perceived risks of heavy alcohol use. 



Percentage of 10th grade students who saw great riik in 
^^ nZ^ol dxinlTin a row, prior to and after Implementation 
of the -Way to Go" program, by school 





Wood County 
High School 
(n = 249) 


Comparison 
school 
(n=182) 


FaU 1992 

(Before "Way to Go") 


19% 


24% 


Spring 1993 
(After **Way to Gol 


29% 


27% 


Percentage difference 


+53% 


+ 13% 


Source: Wood County and Comparison District Student Sun/eys 











Percentage of lOlti grade students who saw greot risk in having five or more drinks in q row, 
prior to and after implementotion of the "Woy to Go" program, by school 



oflOtH^fode 
students 

seeing great 
risk in 5 or 

more drinks 




foll992 
Mon'WayloGo' 



ifkr'WaytoGo' 
I CoM^vffM District 
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After the program, Wood County students appear to see risks associated with alcohol use 
considerably more often than do the students in the neighboring district. Interestingly, 
these differences between Wood County and the comparison distria appear greater than 
the differences in actual alcohol use. Ms. Brown cites these findings as encouraging for 
future behavior. 

These are just a few of the initial findings of the Wood County evaluation. More 
results will be forthcoming from the fall prevalence survey of llth-grade students. These 
findings reflea the first year of implementation of the "Way to Go" program. In year two, 
the drug education coordinator is going to concentrate on obtaining greater uniformity in 
disciplinary policy implementation and teacher participation, particularly among science 
teachers. The comparison distria is now also considering the use of "Way to Go." If it 
decides to adopt the program, however, Wood County will have to find another comparison 
site for the evaluation. 

The brief appendix that follows will provide the reader with additional information 
on where to turn for assistance in developing and implementing a school, district, or agency 
evaluation plan. 
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APPENDIX 



Prevention evaluation guides that address drug prevention specifically include: 

D.C. Gottfredson. DJ. Hawkins, and B. Nederhood, Handbook for Evaluating 
Drug and Alcohol Prevention Programs. A Guide Prepared for Use in the 
Maryland Evaluation of Drug Prevention Programs for High-Risk Youth, Institute 
for Criminal Justice and Criminology, University of Maryland, Oaober 1990. 
While some of the information is intended solely for Maryland-based programs, 
this evaluation guide presents an overview of evaluation issues and a detailed 
discussion of an evaluation method (program development evaluation). It also 
contains an extensive appendix on conducting community needs assessment. 

lOX Associates. Program Evaluation Handbook: Drag Abuse Education, P.O. 
Box 24095. Los Angeles. CA 90024-0095.. This handbook contains an over\'iew 
of evaluation and 20 short instruments on drug use. drug knowledge, refusal and 
decision-making skills, perceived attitudes of friends and family, perceived effects 
of illegal drugs, and perceived seriousness of legal consequences resulting from 
drug use. The handbook contains detailed instruaions on questionnaire 
administration and scoring. 

J. A. Linney and A. Wandersman. Prevention Plus IIL Assessing Alcohol and Other 
Drug Pretention Programs at the School and Community Level. Center for 
Substance Abuse Prevention, U.S. Department of Health and Human Services, 
5600 Fishers Lane, Rockwall II, Rock\ille, MD 20857. This g>iide describes 
evaluation as a four-step process: Step 1: Identify goals and desired outcomes; 
Step 2: Process assessment; Step 3: Outcome assessment; and Step 4: 
Impact assessment. 

D.P. Moberg. Evaluation of Prevention Programs: A Basic Guide for 
Practitioners, by D. Paul Moberg, copyright 1984 by the Board of Regents of the 
University of Wisconsin System for the Wisconsin Clearinghouse. This document 
provides an overview of evaluation that defines key terms, outlines the major 
steps in evaluation, and is particularly suited to evaluations conducted by 
community agencies. 

National or regional surveys cited in this report include: 

Monitoring the Future is a continuing study conducted by the University* of 
Michigan's Institute for Social Research. The questions address attitudes and 
behaviors and are designed for 8th-. 10th-. and 12th-grade students. Although a 
significant number of the questions deal with drugs and alcohol, the majority 
deal with other topics, such as family and school. The questionnaire also 
includes a section of demographic questions, as well as questions alx)ut 
education of parents, religion, educational achievement and aspirations, anc^ 
part-time employment. 
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The National Household Survey on Drug Abuse, conducted by the National 
Institute on Dnig Abuse, contains numerous items to measure frequency and 
recency of use for a variety of substances. There are sections on alcohol, 
sedatives, tranquilizers, stimulants, analgesics, marijuana and hashish, inhalants, 
cocaine, hallucinogens, heroin, drugs in general, smokable methamphetamine 
("ice"), anabolic steroids, trouble widi die law, alcohol and drug problems, 
drug treatment, and perceived risk from drug use. The survey, which is 
intended for persons 12 years or older, is designed to be administered by 
an in-person interviewer. 

The Youth Risk Behavior Survey (YRBS) asks a wide array of questions about 
healdi behavior, including automotive and bicycle safety, swimming, weapons, 
fights, dioughts of suicide, drugs (including cigarettes, alcohol, marijuana, 
cocaine, steroids, and odier drugs) AIDS, sexual activity, dieting, and exercise. 
The survey is designed for junior and senior high school students. 

The National Crime Survey, School Crime Supplement zsVs students 12 years or 
older about crime and victimization at school. 

The Prince George's County Maryland School District Drug Interest Survey asks 
students 10 years and older about their attitudes toward, and use of, tobacco, 
alcohol, and odier drugs. 

The following reports describe and assess a variety of instruments, including instruments 
that could be used in evaluations of drug use prevention programs. 

R.M. Gabriel, J.M. Pollard and J. A. Arter, Surveys of Student Alcohol and Other 
Drug Use: A Consumer's Guide, Northwest Regional Education Laborator)^ 
Portland, OR, 1991: $10.40. 

D.C. Gottfredson, M.A. Harmon, J.A. Lopes, D.L. Stanley, and G.D. Gottfredson, 
Compendium of Instruments to Measure Drug Use and Risk Factors for Drug Use. 
Institute of Criminal Justice and Criminology, University of Maryland, 
1990: $20.00. 

C.L. Kumpfer, G.H. Shur, J.G. Ross, and J.L. Librett, Measurement in Prevention, A 
Manual for the Selection and Use of Instruments to Evaluate Prevention Programs, 
Office for Substance Abuse Prevention, U.S. Department of Healdi and 
Human Services, Rockville, MD (this report is scheduled to be published in 
October 1992). 
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Additional information on topics addressed in this handbook as well as specific assistance 
in conducting evaluations may be obtained by contacting the U.S. Department of 
Education Regional Centers for Drug-Free Schools and Communities. They include: 



Northeast Regional Center fur Drug-Free 

Schools and Communities 
12 Overton Avenue 
Sayville, NY 11782 
(516) 589-7894 

Southeast Regional Center for Drug-Free Schools 

and Communities 
Spencerian Office Plaza 
University of Louisville 
Louisville, KY 40292 
(502) 588-0052 

Midwest Regional Center for Dmg-Free Schools 

and Communities 
1900 Spring Road 
Oak Brook, IL 60521 
(708) 571-4710 

Southwest Regional Center for Drug-Free Schools 

and Communities 
The University of Oklahoma 
555 Constitution, Suite 138 
Norman, OK 73037-0005 
(800) 234-7972 

Western Regional Center for Drug-Free Schools 

and Communities 
101 SW. Main Street. Suite 500 
Portland. OR 97204 
(503) 275-9480 



ERLC 
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